The Photon iMessage sidecar needs node_modules under
plugins/platforms/photon/sidecar/, but hosted/managed images keep the
whole install tree under an immutable /opt/hermes — every install and
self-heal path (setup CLI, stale-deps reinstall, cold install) died on
EROFS, and hosted users have no shell to work around it.
Three-layer fix, mirroring the WhatsApp bridge resolver pattern:
1. Bake the deps into the image. The Dockerfile now runs npm ci for the
sidecar in the layer-cached dependency stage (deterministic installs
from the committed lockfile; the postinstall spectrum-ts patch runs
at build time). Hosted happy path needs no runtime install at all.
2. New sidecar_paths.resolve_sidecar_dir() decides where the sidecar
runs from: PHOTON_SIDECAR_DIR override > writable source dir (dev
installs, unchanged) > read-only dir with baked fresh deps (managed
image) > mirror to $HERMES_HOME/photon/sidecar (writable data
volume) when deps are missing or stale in a read-only tree. The
mirror refreshes changed source files on image updates while
keeping node_modules, so the existing lockfile-staleness self-heal
works there.
3. connect() can now cold-install: _start_sidecar() runs the bounded
npm ci bootstrap when node_modules is missing instead of raising
immediately, and check_requirements() reports available when a
self-install is possible (npm present + writable resolved dir) so
the gateway actually creates the adapter on hosted instances. A
failed bootstrap still raises the actionable error, which connect()
surfaces as the retryable SIDECAR_FAILED fatal state on the
dashboard.
Tests: resolver decision table (env override, in-place, mirror,
refresh, fail-open), cold-install lifecycle paths, and a Dockerfile
contract test guarding the baked-deps + no-chown invariants.
Fixes NS-606.
`PhotonAdapter._start_sidecar` is `async`, but it ran the Spectrum
mixed-attachment patch script with a bare `subprocess.run(...)`: it spawns
node and *waits* for it, with `timeout=10`. Executed inline that holds the
shared gateway event loop for the whole window, so no other platform's
messages, heartbeats, or sessions are serviced until it returns.
The same function already establishes this exact invariant twenty lines
above, where the stale-dependency reinstall hops to a worker thread:
# Runs off the event loop so a cold install can't freeze every other
# platform's traffic.
if _sidecar_deps_stale():
await asyncio.to_thread(_reinstall_sidecar_deps)
The patch spawn never got the same treatment. It is not startup-only
either — `_start_sidecar` is called from `connect()`, which takes
`is_reconnect`, so an ordinary Photon reconnect (network blip, sidecar
death) re-runs it and stalls a live gateway that is actively serving
Discord/Telegram/Slack traffic.
Dispatch it via `asyncio.to_thread` like its sibling. Same off-the-loop
class as the inbound-image decision (#66688) and the cron-fire verifier.
Adds a regression test asserting the spawn executes on a worker thread
rather than the loop thread.
`_reap_stale_sidecar` is `async`, but it identified the processes holding
the sidecar port with two blocking helpers called inline:
* `_find_listener_pids` -> `subprocess.run(["lsof", ...], timeout=5.0)`
* `_pid_is_sidecar` -> `subprocess.run(["ps", ...], timeout=5.0)`, once
per candidate pid
so the inspection can hold the shared gateway loop for 5 + 5·N seconds
while nothing else on it is serviced. It only runs once the /healthz probe
finds something already listening — the orphaned-sidecar recovery path —
and `_reap_stale_sidecar` is awaited from `_start_sidecar`, which runs on
every reconnect (`connect(is_reconnect=True)`). The stall therefore lands
on a live gateway that is still serving every other platform, right when a
crashed sidecar has already left an orphan behind.
Move the whole inspection to one `asyncio.to_thread` hop (one hop rather
than N+1 round trips). The reaping semantics are untouched: SIGTERM for
verified orphans, SIGKILL escalation, and both foreign-listener
RuntimeErrors behave exactly as before.
Same off-the-loop class as the inbound-image decision (#66688) and the
cron-fire verifier.
Adds a regression test asserting both the lsof lookup and the per-pid ps
check execute on a worker thread rather than the loop thread.
Addresses teknium1 review on #50983:
- cli.py's `hermes photon status` and adapter.py's _start_sidecar() still
used the old node_modules/-existence check while check_requirements()
had moved to a spectrum-ts content check. Extracted the check into a
shared sidecar_deps_installed(), used by all three, so an
empty/partial node_modules/ (aborted npm install) is rejected
consistently instead of only in check_requirements().
- _install_sidecar()'s success-path _NPM_ERROR_LOG.unlink() only caught
FileNotFoundError, so a PermissionError/OSError on a locked file would
propagate. Broadened to OSError.
- npm stderr was truncated only when read back in check_requirements();
an unbounded stderr was written to disk on every failed install. Now
truncated to _NPM_ERROR_LOG_MAX_CHARS before write_text().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A hard gateway exit (crash, SIGKILL, supervisor restart) left the
detached Node sidecar running with a token the next gateway run doesn't
know, so it could never be told to /shutdown. Every replacement spawn
then died on EADDRINUSE, failing each 30→300s reconnect attempt while
the orphan kept consuming the inbound gRPC stream.
Two layers:
- Lifetime binding: the adapter now holds the sidecar's stdin as a
pipe, and the sidecar (PHOTON_SIDECAR_WATCH_STDIN=1) shuts down on
stdin EOF — fired by the OS on any parent death, including SIGKILL.
- Startup reaping: before spawning, the adapter probes the port and
terminates a stale listener, but only after verifying its command
line is a Photon sidecar; a foreign listener raises a clear error
instead of being signalled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
