mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-29 18:46:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

fix(photon): rework zombie-stream watchdog for spectrum-ts 8 with strict probe semantics

Browse source 
Maintainer rework of #45580 (issue #54036) on top of the contributor's
cherry-pick, which targeted spectrum-ts 3.1.0 while main pins 8.0.0:

Sidecar (primary detection, new):
- stream-staleness.mjs: pure decision rules, executable under node.
  * classifyProbeRejection: only a not-found-shaped rejection of the
    synthetic-id read counts as a completed round-trip (ALIVE); any other
    rejection is INCONCLUSIVE — never alive. The original /probe treated
    ANY rejection as alive, which was too loose.
  * shouldProbe: probe only after 10+ min of stream silence (configurable
    via PHOTON_STREAM_SILENCE_PROBE_MS; <=0 disables) with a cooldown.
  * isZombieSuspect: zombie only on silence past threshold AND a
    probe-proven live channel. Silence alone NEVER degrades (shared lines
    can be quiet for hours); inconclusive probes NEVER degrade (network
    may be down — the iterator will throw and the re-subscribe loop
    recovers on its own).
- index.mjs: track last inbound-iterator yield (noteInboundYield), run a
  30s watchdog tick, and on a confirmed zombie feed markStreamDegraded ->
  the existing exit-75 restart path. /healthz gains a stream.staleness
  block (silentForMs, threshold, lastProbeOutcome, zombieSuspected).
  /probe reworked to strict semantics: 200 only on a proven round-trip,
  503 with outcome hung|inconclusive otherwise.

Adapter (second layer, reworked):
- _probe_once returns tri-state alive|hung|inconclusive; only a hung
  sidecar HTTP call counts toward the respawn counter — inconclusive
  resets nothing and triggers nothing.
- default probe_interval_seconds 60 -> 600 (conservative; avoid restart
  storms on quiet lines).
- _monitor_sidecar_health surfaces zombieSuspected from /healthz as a
  warning; the fatal UPSTREAM_STREAM_DEGRADED path is unchanged and fires
  when the sidecar escalates.

Tests: test_zombie_stream_watchdog.py executes the real node decision
module and drives the adapter against mocked /healthz responses;
test_presence_watchdog.py updated for the tri-state probe.

Also adds contributor mappings for nickkarhan (#53283) and vaibhavjnf
(#45580).
This commit is contained in:
Teknium 2026-07-28 12:35:48 -07:00
parent 87fe75fde4
commit dcace573da
6 changed files with 656 additions and 72 deletions
Download patch file Download diff file Expand all files Collapse all files

115
plugins/platforms/photon/adapter.py
View file

@ -338,6 +338,17 @@ def _first_set(*values: Any) -> Any:
return None
def _is_timeout_error(exc: BaseException) -> bool:
"""True when *exc* indicates the request timed out (call hung)."""
if isinstance(exc, (asyncio.TimeoutError, TimeoutError)):
return True
if HTTPX_AVAILABLE:
timeout_exc = getattr(httpx, "TimeoutException", None)
if timeout_exc is not None and isinstance(exc, timeout_exc):
return True
return "timeout" in type(exc).__name__.lower()
def check_requirements() -> bool:
"""Return True when both Python deps and the Node sidecar are available."""
if not HTTPX_AVAILABLE:
@ -668,11 +679,17 @@ class PhotonAdapter(BasePlatformAdapter):
# Presence watchdog. spectrum-ts only reconnects when its inbound
# iterator throws or ends; a half-open ("zombie") gRPC socket makes the
# iterator hang forever (no error, no end), so inbound silently dies
# until the sidecar is restarted. We periodically drive a cheap upstream
# unary read via the sidecar's /probe endpoint; repeated failures mean
# the stream is dead -> respawn the sidecar (fresh Spectrum() = fresh
# gRPC stream). A successful probe is also a real round-trip that keeps
# the channel warm, helping prevent the zombie forming at all.
# until the sidecar is restarted. The sidecar owns primary zombie
# detection (stream-staleness + upstream probe -> degraded -> exit 75;
# surfaced here via /healthz in _monitor_sidecar_health). This adapter-
# side watchdog is a conservative second layer that only respawns the
# sidecar when the sidecar's own HTTP loop stops responding (probe
# HTTP call hangs) — an "inconclusive" probe (sidecar answered but
# could not prove upstream liveness) NEVER counts toward a respawn:
# the network may simply be down, and restarting cannot fix that.
# Thresholds are deliberately conservative (10 min interval) — shared
# lines can be legitimately quiet for hours, so we never restart on
# silence alone.
# Behavioural settings -> config.yaml (extra), bridged to env.
# Use _first_set (not ``or``) so an explicit 0 is honored — ``0 or X``
# would silently fall through to the default and you could never
@ -682,7 +699,7 @@ class PhotonAdapter(BasePlatformAdapter):
extra.get("probe_interval_seconds"),
os.getenv("PHOTON_PROBE_INTERVAL_SECONDS"),
),
60.0,
600.0,
)
self._probe_timeout = _coerce_float(
_first_set(
@ -1029,7 +1046,26 @@ class PhotonAdapter(BasePlatformAdapter):
continue
stream = data.get("stream") if isinstance(data, dict) else None
if not isinstance(stream, dict) or stream.get("ok") is not False:
if not isinstance(stream, dict):
continue
# Surface the sidecar's zombie-stream staleness state early: a
# suspected half-open stream (silence past threshold + probe-proven
# connectivity) is worth a loud log line even before the sidecar's
# degraded->exit-75 path fires.
staleness = stream.get("staleness")
if (
isinstance(staleness, dict)
and staleness.get("zombieSuspected") is True
):
logger.warning(
"[photon] sidecar reports suspected zombie stream"
" (silentForMs=%s, lastProbeOutcome=%s)",
staleness.get("silentForMs"),
staleness.get("lastProbeOutcome"),
)
if stream.get("ok") is not False:
continue
state = str(stream.get("state") or "unknown")
@ -1712,15 +1748,24 @@ class PhotonAdapter(BasePlatformAdapter):
self._last_upstream_activity = time.monotonic()
self._probe_failures = 0
async def _probe_once(self) -> bool:
"""Drive one upstream liveness probe via the sidecar's ``/probe``.
async def _probe_once(self) -> str:
"""Drive one liveness probe via the sidecar's ``/probe``.
Returns True if the sidecar confirmed a live gRPC round-trip within
the timeout, False on timeout/error (a likely zombie stream).
Returns a strict tri-state verdict:
- ``"alive"`` — the sidecar completed a real upstream gRPC
round-trip (HTTP 200). Only this counts as proof of liveness.
- ``"hung"`` — the probe HTTP call itself timed out: the
sidecar's event loop is unresponsive. Counts toward respawn.
- ``"inconclusive"`` — anything else (503 from the sidecar's strict
probe, connection refused, transport error). NEVER counts as alive
and NEVER counts toward a respawn: a rejected upstream probe may
just mean the network is down, and a dead sidecar process is the
supervisor's job, not ours.
"""
client = self._http_client
if client is None:
return False
return "inconclusive"
url = f"http://{self._sidecar_bind}:{self._sidecar_port}/probe"
try:
resp = await client.post(
@ -1728,10 +1773,15 @@ class PhotonAdapter(BasePlatformAdapter):
headers={"X-Hermes-Sidecar-Token": self._sidecar_token},
timeout=self._probe_timeout,
)
except asyncio.CancelledError:
raise
except Exception as e:
logger.debug("[photon] probe transport error: %s", e)
return False
return resp.status_code == 200
if _is_timeout_error(e):
logger.debug("[photon] probe HTTP call hung: %s", e)
return "hung"
logger.debug("[photon] probe transport error (inconclusive): %s", e)
return "inconclusive"
return "alive" if resp.status_code == 200 else "inconclusive"
async def _respawn_sidecar(self, reason: str) -> None:
"""Tear down and restart the sidecar to recover a dead gRPC stream.
@ -1769,14 +1819,18 @@ class PhotonAdapter(BasePlatformAdapter):
logger.info("[photon] presence watchdog: sidecar respawned, gRPC stream renewed")
async def _presence_watchdog(self) -> None:
"""Periodically confirm the upstream gRPC stream is alive.
"""Periodically confirm the sidecar (and its stream) is responsive.
spectrum-ts only recovers when its inbound iterator throws or ends; a
half-open ("zombie") socket makes it hang forever with no error, so
inbound silently dies. We probe the channel on an interval (skipping
the probe when natural inbound traffic already proved liveness within
the window). After ``_probe_max_failures`` consecutive failed probes we
respawn the sidecar to force a fresh stream.
The sidecar owns primary zombie-stream detection (silence tracking +
upstream probe -> degraded /healthz -> exit 75). This loop is the
adapter's conservative second layer: it probes on a long interval,
skips the probe when natural inbound traffic already proved liveness,
and only counts a *hung* probe (the sidecar HTTP call itself timing
out) toward a respawn. Inconclusive probes — the sidecar answered but
couldn't prove upstream liveness — reset nothing and trigger nothing:
the network may just be down, and restarting can't fix that. After
``_probe_max_failures`` consecutive hung probes we respawn the sidecar
to force a fresh stream.
"""
# Stagger the first probe so a fleet of restarts doesn't synchronize,
# and so a freshly-started sidecar isn't probed before it's warm.
@ -1791,19 +1845,26 @@ class PhotonAdapter(BasePlatformAdapter):
await asyncio.sleep(self._probe_interval - idle)
continue
alive = await self._probe_once()
if alive:
verdict = await self._probe_once()
if verdict == "alive":
self._note_upstream_activity()
else:
elif verdict == "hung":
# Only a hung sidecar HTTP loop counts toward respawn —
# strict success-only-on-probe-OK semantics mean an
# inconclusive probe is never evidence in either direction.
self._probe_failures += 1
logger.warning(
"[photon] presence probe failed (%d/%d)",
"[photon] presence probe hung (%d/%d)",
self._probe_failures, self._probe_max_failures,
)
if self._probe_failures >= self._probe_max_failures:
await self._respawn_sidecar(
f"{self._probe_failures} consecutive probe failures"
f"{self._probe_failures} consecutive hung probes"
)
else:
logger.debug(
"[photon] presence probe inconclusive; taking no action"
)
except asyncio.CancelledError:
raise
except Exception:

191
plugins/platforms/photon/sidecar/index.mjs
View file

@ -67,6 +67,11 @@ import crypto from "node:crypto";
import { once } from "node:events";
import { patchSpectrumTs } from "./patch-spectrum-mixed-attachments.mjs";
import { chooseSendFormat } from "./send-format.mjs";
import {
classifyProbeRejection,
shouldProbe,
isZombieSuspect,
} from "./stream-staleness.mjs";
const projectId = process.env.PHOTON_PROJECT_ID;
const projectSecret = process.env.PHOTON_PROJECT_SECRET;
@ -93,6 +98,22 @@ const STREAM_DEGRADED_RESTART_MS =
Number(process.env.PHOTON_STREAM_DEGRADED_RESTART_MS) || 90 * 1000;
const STREAM_INTERRUPTED_DEGRADE_COUNT =
Number(process.env.PHOTON_STREAM_INTERRUPTED_DEGRADE_COUNT) || 3;
// Zombie-stream (half-open gRPC) watchdog. The inbound iterator can hang
// forever without erroring; we track when it last yielded and, once it has
// been silent past this threshold, drive a cheap authenticated probe. Silence
// alone NEVER degrades the stream (shared lines can be quiet for hours) and a
// rejected probe is INCONCLUSIVE, never proof either way — degradation
// requires silence past the threshold plus a probe-proven live channel.
// A non-positive threshold disables the watchdog.
const STREAM_SILENCE_PROBE_MS = (() => {
const raw = Number(process.env.PHOTON_STREAM_SILENCE_PROBE_MS);
return Number.isFinite(raw) ? raw : 10 * 60 * 1000;
})();
const STREAM_PROBE_COOLDOWN_MS =
Number(process.env.PHOTON_STREAM_PROBE_COOLDOWN_MS) || 2 * 60 * 1000;
const STREAM_PROBE_TIMEOUT_MS =
Number(process.env.PHOTON_STREAM_PROBE_TIMEOUT_MS) || 10 * 1000;
const STREAM_WATCHDOG_TICK_MS = 30 * 1000;
const streamHealth = {
state: "starting",
@ -104,6 +125,33 @@ const streamHealth = {
};
let streamRestartTimer = null;
// Zombie-watchdog runtime state (see stream-staleness.mjs for the rules).
const staleness = {
lastInboundAt: Date.now(),
lastProbeAt: 0,
lastProbeOutcome: null, // "alive" | "inconclusive" | null
zombieSuspected: false,
};
function noteInboundYield() {
staleness.lastInboundAt = Date.now();
staleness.zombieSuspected = false;
}
function stalenessSnapshot(now) {
return {
lastInboundAt: new Date(staleness.lastInboundAt).toISOString(),
silentForMs: now - staleness.lastInboundAt,
silenceThresholdMs: STREAM_SILENCE_PROBE_MS,
lastProbeAt:
staleness.lastProbeAt > 0
? new Date(staleness.lastProbeAt).toISOString()
: null,
lastProbeOutcome: staleness.lastProbeOutcome,
zombieSuspected: staleness.zombieSuspected,
};
}
function streamHealthSnapshot() {
const now = Date.now();
const degradedForMs =
@ -117,6 +165,7 @@ function streamHealthSnapshot() {
lastIssueAt: streamHealth.lastIssueAt,
lastIssue: streamHealth.lastIssue,
issueCount: streamHealth.issueCount,
staleness: stalenessSnapshot(now),
};
}
@ -599,6 +648,8 @@ function inboundStreamErrorMessage(e) {
for await (const [space, message] of app.messages) {
backoff = 1000; // healthy traffic — reset
markStreamHealthy();
// Every yield — any direction — proves the inbound stream is live.
noteInboundYield();
// Only forward inbound messages (ignore our own outbound echoes).
if (message && message.direction && message.direction !== "inbound") {
continue;
@ -623,6 +674,99 @@ function inboundStreamErrorMessage(e) {
}
})();
// ---------------------------------------------------------------------------
// Zombie-stream watchdog: detect a half-open gRPC stream the SDK can't see.
//
// The inbound iterator above can hang forever on a half-open socket — no
// error, no end — so the re-subscribe loop never fires and classifyStreamLog
// never sees a "[spectrum.stream]" line. We track the iterator's last yield
// (noteInboundYield) and, once it has been silent past
// STREAM_SILENCE_PROBE_MS, drive a cheap unary read (space.getMessage on a
// synthetic id) over the same authenticated channel. Decision rules (see
// stream-staleness.mjs):
// probe proves connectivity while the stream is silent -> the stream itself
// is the dead part -> markStreamDegraded -> existing exit-75 restart path.
// probe inconclusive (rejected/hung) -> do NOTHING: the network may just be
// down, and in that case the iterator eventually throws and the
// re-subscribe loop recovers on its own. Never restart on silence alone —
// shared lines can be legitimately quiet for hours.
async function probeUpstream() {
if (typeof app?.stop !== "function") {
return { alive: false, hung: false, reason: "spectrum app not constructed" };
}
const probeId =
PROBE_MSG_PREFIX + Date.now() + "-" + Math.random().toString(36).slice(2);
let timer = null;
const timeout = new Promise((resolve) => {
timer = setTimeout(
() => resolve({ alive: false, hung: true, reason: "probe timed out" }),
STREAM_PROBE_TIMEOUT_MS
);
timer.unref();
});
const attempt = (async () => {
try {
const im = imessage(app);
const space = await im.space.get(PROBE_SPACE_ID);
await space.getMessage(probeId);
// Resolving for a synthetic id is unexpected but is still a completed
// round-trip: the channel is alive.
return { alive: true, hung: false, reason: "round-trip completed" };
} catch (e) {
const verdict = classifyProbeRejection(e);
return { alive: verdict.alive, hung: false, reason: verdict.reason };
}
})();
const outcome = await Promise.race([attempt, timeout]);
if (timer) clearTimeout(timer);
staleness.lastProbeAt = Date.now();
staleness.lastProbeOutcome = outcome.alive ? "alive" : "inconclusive";
return outcome;
}
let watchdogProbeInFlight = false;
async function zombieWatchdogTick() {
if (watchdogProbeInFlight) return;
const now = Date.now();
const silentForMs = now - staleness.lastInboundAt;
if (
!shouldProbe(
silentForMs,
STREAM_SILENCE_PROBE_MS,
now - staleness.lastProbeAt,
STREAM_PROBE_COOLDOWN_MS
)
) {
return;
}
watchdogProbeInFlight = true;
try {
const outcome = await probeUpstream();
if (isZombieSuspect(silentForMs, STREAM_SILENCE_PROBE_MS, outcome)) {
staleness.zombieSuspected = true;
const reason =
`inbound stream silent for ${silentForMs}ms while an upstream probe ` +
"succeeded — half-open (zombie) gRPC stream suspected";
console.error("photon-sidecar: " + reason);
markStreamDegraded(reason);
}
// Inconclusive: deliberately no action (see block comment above).
} catch (e) {
console.error(
"photon-sidecar: zombie watchdog tick failed: " +
(e && e.message ? e.message : String(e))
);
} finally {
watchdogProbeInFlight = false;
}
}
if (STREAM_SILENCE_PROBE_MS > 0) {
const watchdogTimer = setInterval(zombieWatchdogTick, STREAM_WATCHDOG_TICK_MS);
watchdogTimer.unref();
}
// ---------------------------------------------------------------------------
// HTTP control + inbound server (loopback only).
@ -842,37 +986,24 @@ const server = http.createServer(async (req, res) => {
}
if (req.url === "/probe") {
// Upstream liveness probe. Drives a cheap unary read over the SAME gRPC
// channel the inbound stream uses. On a live channel this round-trips
// fast (the server returns "not found" for the synthetic id, which the
// SDK surfaces as a thrown error — that's a SUCCESS for our purposes:
// the wire is alive). On a half-open zombie socket the call hangs and
// the caller's timeout fires, so the adapter can respawn us. It sends
// nothing to any user and creates no chat (space.get is local; only the
// message read touches the network).
if (typeof app?.stop !== "function") {
// app failed to construct — definitely not live.
return serverError(res);
}
try {
const im = imessage(app);
const space = await im.space.get(PROBE_SPACE_ID);
const probeId = PROBE_MSG_PREFIX + Date.now() + "-" + Math.random().toString(36).slice(2);
try {
await space.getMessage(probeId);
} catch {
// Expected: the synthetic id doesn't exist. Reaching here means the
// unary call completed a round-trip — the channel is ALIVE.
}
return ok(res, { alive: true });
} catch (e) {
// space.get is local; an error here is unexpected but means we can't
// even build a probe — report not-alive so the adapter recycles us.
console.error(
"photon-sidecar: probe failed: " +
(e && e.message ? e.message : String(e))
);
return serverError(res);
// channel the inbound stream uses. STRICT semantics: only a completed
// round-trip (resolution, or a not-found rejection for our synthetic
// id) counts as alive; any other rejection or a timeout is
// INCONCLUSIVE — never reported as alive.
const outcome = await probeUpstream();
if (outcome.alive) {
return ok(res, { alive: true, outcome: "alive" });
}
res.statusCode = 503;
res.setHeader("Content-Type", "application/json");
return res.end(
JSON.stringify({
ok: false,
alive: false,
outcome: outcome.hung ? "hung" : "inconclusive",
reason: outcome.reason,
})
);
}
if (req.url === "/shutdown") {
ok(res, {});

80
plugins/platforms/photon/sidecar/stream-staleness.mjs Normal file
View file

@ -0,0 +1,80 @@
// Pure decision helpers for the zombie-stream (half-open gRPC) watchdog.
//
// spectrum-ts only reconnects when its inbound async iterator throws or ends.
// A half-open ("zombie") socket makes the iterator hang forever — no error,
// no end — so inbound silently dies while /healthz still looks fine. The
// watchdog in index.mjs tracks the last time the inbound iterator yielded and,
// once the stream has been silent past a conservative threshold, drives a
// cheap authenticated unary read over the same channel. STRICT semantics:
//
// - probe resolves, or rejects with a not-found-shaped error for our
// synthetic id -> ALIVE (the wire round-tripped)
// - probe rejects any other way (UNAVAILABLE, DEADLINE_EXCEEDED, network
// down, ...) -> INCONCLUSIVE — never treated as alive, and
// never treated as zombie-proof either
//
// A zombie is only declared when the stream is silent past the threshold AND
// a probe proves connectivity (the wire works but the stream is deaf). Silence
// alone NEVER degrades the stream: shared lines can be legitimately quiet for
// hours. Inconclusive probes NEVER degrade it either: the network may simply
// be down, and in that case the iterator will eventually throw and the
// existing re-subscribe loop recovers on its own.
//
// These helpers are pure (no SDK, no timers) so tests can execute them under
// node — see tests/plugins/platforms/photon/test_zombie_stream_watchdog.py.
// gRPC NOT_FOUND is code 5; SDKs also surface it as "not found" / "NotFound"
// message text. Anything not clearly not-found is inconclusive.
const NOT_FOUND_RE = /not[\s_-]?found/i;
/**
* Classify the rejection of the synthetic-id probe read.
*
* @param {unknown} err error thrown by `space.getMessage(<synthetic id>)`
* @returns {{alive: boolean, inconclusive: boolean, reason: string}}
*/
export function classifyProbeRejection(err) {
const code = err && typeof err === "object" ? err.code : undefined;
const message =
err && typeof err === "object" && err.message
? String(err.message)
: String(err);
if (code === 5 || code === "notFound" || NOT_FOUND_RE.test(message)) {
// Expected: the synthetic id doesn't exist. The unary call completed a
// round-trip, so the channel is provably alive.
return { alive: true, inconclusive: false, reason: "not-found round-trip" };
}
// Anything else (UNAVAILABLE, DEADLINE_EXCEEDED, TLS, auth, ...) does NOT
// prove liveness — and doesn't prove a zombie either.
return { alive: false, inconclusive: true, reason: message };
}
/**
* Should the watchdog probe at all this tick?
*
* @param {number} silentForMs ms since the inbound iterator last yielded
* @param {number} thresholdMs silence threshold (<= 0 disables the watchdog)
* @param {number} sinceLastProbeMs ms since the previous probe attempt
* @param {number} probeCooldownMs min spacing between probe attempts
* @returns {boolean}
*/
export function shouldProbe(silentForMs, thresholdMs, sinceLastProbeMs, probeCooldownMs) {
if (!(thresholdMs > 0)) return false;
if (silentForMs < thresholdMs) return false;
return sinceLastProbeMs >= probeCooldownMs;
}
/**
* Final classification: zombie only on silence past threshold + probe-proven
* connectivity. Never on silence alone, never on an inconclusive probe.
*
* @param {number} silentForMs ms since the inbound iterator last yielded
* @param {number} thresholdMs silence threshold (<= 0 disables the watchdog)
* @param {{alive: boolean}} probeOutcome
* @returns {boolean}
*/
export function isZombieSuspect(silentForMs, thresholdMs, probeOutcome) {
if (!(thresholdMs > 0)) return false;
if (silentForMs < thresholdMs) return false;
return probeOutcome != null && probeOutcome.alive === true;
}