Maintainer rework of #45580 (issue #54036) on top of the contributor's
cherry-pick, which targeted spectrum-ts 3.1.0 while main pins 8.0.0:
Sidecar (primary detection, new):
- stream-staleness.mjs: pure decision rules, executable under node.
* classifyProbeRejection: only a not-found-shaped rejection of the
synthetic-id read counts as a completed round-trip (ALIVE); any other
rejection is INCONCLUSIVE — never alive. The original /probe treated
ANY rejection as alive, which was too loose.
* shouldProbe: probe only after 10+ min of stream silence (configurable
via PHOTON_STREAM_SILENCE_PROBE_MS; <=0 disables) with a cooldown.
* isZombieSuspect: zombie only on silence past threshold AND a
probe-proven live channel. Silence alone NEVER degrades (shared lines
can be quiet for hours); inconclusive probes NEVER degrade (network
may be down — the iterator will throw and the re-subscribe loop
recovers on its own).
- index.mjs: track last inbound-iterator yield (noteInboundYield), run a
30s watchdog tick, and on a confirmed zombie feed markStreamDegraded ->
the existing exit-75 restart path. /healthz gains a stream.staleness
block (silentForMs, threshold, lastProbeOutcome, zombieSuspected).
/probe reworked to strict semantics: 200 only on a proven round-trip,
503 with outcome hung|inconclusive otherwise.
Adapter (second layer, reworked):
- _probe_once returns tri-state alive|hung|inconclusive; only a hung
sidecar HTTP call counts toward the respawn counter — inconclusive
resets nothing and triggers nothing.
- default probe_interval_seconds 60 -> 600 (conservative; avoid restart
storms on quiet lines).
- _monitor_sidecar_health surfaces zombieSuspected from /healthz as a
warning; the fatal UPSTREAM_STREAM_DEGRADED path is unchanged and fires
when the sidecar escalates.
Tests: test_zombie_stream_watchdog.py executes the real node decision
module and drives the adapter against mocked /healthz responses;
test_presence_watchdog.py updated for the tri-state probe.
Also adds contributor mappings for nickkarhan (#53283) and vaibhavjnf
(#45580).
Follow-up to the URL markdown fix: extract the /send builder decision into
sidecar/send-format.mjs and rewrite test_url_send_path.py to execute the real
module under node (format+text in -> chosen builder out) instead of regex-
grepping index.mjs source, which is a banned test pattern in this repo.
spectrum-ts's live-stream consumer (consumeLive in spectrum-ts 3.x) only
reconnects when its inbound async iterator throws or ends. A half-open
("zombie") gRPC socket — where the TCP connection stays ESTABLISHED but the
peer is gone (NAT idle-timeout, network blip, laptop sleep) — makes the
iterator hang forever: no error, no end. The SDK exposes no gRPC keepalive
knob (createClient takes only {address, tls, token}; grpc.keepalive_time_ms
defaults to -1 = pings off), so the inbound stream silently dies and stays
dead until the gateway is restarted. Symptom: the agent's iMessage line goes
"online but deaf" — Photon's cloud-side fallback answers users with "the agent
isn't online right now" and inbound never reaches the gateway.
Fix, entirely in the code we own (no SDK fork):
- Sidecar gains a POST /probe endpoint that drives a cheap unary read
(space.getMessage on a synthetic id) over the SAME gRPC channel the inbound
stream uses. A live channel round-trips in ms (server returns not-found,
which is success for liveness); a zombie hangs. It sends nothing to any user
and creates no chat (space.get is local in shared/dedicated mode; only the
message read touches the wire).
- The adapter runs a presence watchdog: it probes on an interval, skips the
probe when natural inbound traffic already proved liveness within the
window, and after N consecutive failed probes respawns the sidecar — a fresh
Spectrum() re-subscribes the stream and re-registers presence. Successful
probes double as application-level keepalive, helping prevent the zombie
from forming at all. Respawn is lock-guarded against double-spawn and the
watchdog is torn down cleanly on disconnect.
Behavioural settings live in config.yaml (extra), bridged to env per the
.env-is-secrets-only convention:
probe_interval_seconds (60), probe_timeout_seconds (10),
probe_max_failures (3). A non-positive interval disables the watchdog.
Tests: tests/plugins/platforms/photon/test_presence_watchdog.py covers config
resolution, the disable switch, probe alive/dead(500)/timeout/no-client, the
core N-failures->one-respawn detection, success-resets-failures, stop-then-
start respawn ordering, and lock-guarding — all without spawning Node or
hitting the network.
Contributed by Vaibhav Sharma (X: @vabbyshabby).
The `clarify` tool's multiple-choice prompts flattened to a numbered text
list on Photon/iMessage, even though iMessage has a native poll bubble and
spectrum-ts already exposes it via the `poll()` content builder. Two gaps
caused the flattening:
* Outbound: the sidecar only had `/send` (text); there was no way to send
a poll, so the base adapter's numbered-text fallback was used.
* Inbound: `normalizeContent()` handled only text/attachment/voice, so a
poll vote (`poll_option`) was dropped on the floor ("[Photon content
type not handled: poll_option]") and never resolved the clarify.
Fix, end to end:
* Sidecar: import `poll` from spectrum-ts; add a `/send-poll` route
(`space.send(poll(title, ...options))`); serialize inbound `poll_option`
(the vote: chosen title + selected bool) and `poll` content in
`normalizeContent()`.
* Adapter: override `send_clarify` — for choices, send a native poll via
`_sidecar_send_poll` and call `mark_awaiting_text` so the gateway's
existing pending-clarify text-intercept resolves the answer; open-ended
clarifies keep the plain-text path. Inbound `poll_option` selections are
dispatched as a plain-text MessageEvent carrying the chosen option
(deselections / empty votes are dropped). If the poll send fails (an
older sidecar without `/send-poll`, or a send error) it falls back to the
numbered-text clarify, so nothing regresses on a half-upgraded restart.
No new model tool, no new env var, no core change — the capability lives at
the platform edge. The poll vote reuses the existing clarify text-intercept
resolution path, so no new gateway resolution mechanism is introduced.
Tests: tests/plugins/platforms/photon/test_poll_clarify.py — inbound vote ->
choice text, deselection/empty-vote dropped, send_clarify sends a poll +
enables text-capture, open-ended stays text, and poll-failure falls back to
the text list. Full photon suite green.
Contributed by Vaibhav Sharma (X: @vabbyshabby).
Maintainer follow-up to the #51193 salvage:
- _send_with_retry: permanent classes (auth_or_config, target_not_allowed)
now short-circuit BEFORE the unconditional plain-text fallback resend,
including when a retry attempt surfaces one — no more double-sends of
permanently-failing requests.
- sidecar classifySidecarError: new structured code target_not_allowed for
Spectrum's 'Target not allowed for this project' AuthenticationError
(shared/free-tier lines cannot initiate outbound sends to new targets).
Classification applies to every handler sharing the catch-all
serverError path (/send, /send-attachment, /react, /typing, ...).
- _standalone_send now parses the structured error body too (it reads
sidecar responses independently of _sidecar_call) and returns
error_class/retryable alongside the message.
- target_not_allowed maps to a canonical user-facing message in both
paths; raw upstream error text never leaks through the structured code.
Closes the actionable halves of #50971, #51897, #52794.
Photon's Node sidecar intentionally hides raw handler exceptions, but the Python adapter still needs a safe failure class and retryability bit so delivery retries do not collapse into an opaque generic 500.
Constraint: Sidecar responses must not leak raw stack traces or private exception text
Rejected: Retry every internal sidecar error | masks permanent auth/config failures
Confidence: high
Scope-risk: narrow
Directive: Keep sidecar error text generic; extend safe error classes instead of exposing raw SDK failures
Tested: uv run --with pytest-timeout pytest tests/plugins/platforms/photon/test_overflow_recovery.py -q
Tested: uv run --with pytest-timeout pytest tests/plugins/platforms/photon -q
Tested: uv run ruff check plugins/platforms/photon/adapter.py tests/plugins/platforms/photon/test_overflow_recovery.py
Tested: python3 -m py_compile plugins/platforms/photon/adapter.py tests/plugins/platforms/photon/test_overflow_recovery.py
Tested: node --check plugins/platforms/photon/sidecar/index.mjs
Tested: git diff --check
Tested: python3 scripts/check-windows-footguns.py --diff origin/main
Not-tested: Live Photon/Spectrum delivery against a real iMessage account
Related: #50971
`PhotonAdapter._start_sidecar` is `async`, but it ran the Spectrum
mixed-attachment patch script with a bare `subprocess.run(...)`: it spawns
node and *waits* for it, with `timeout=10`. Executed inline that holds the
shared gateway event loop for the whole window, so no other platform's
messages, heartbeats, or sessions are serviced until it returns.
The same function already establishes this exact invariant twenty lines
above, where the stale-dependency reinstall hops to a worker thread:
# Runs off the event loop so a cold install can't freeze every other
# platform's traffic.
if _sidecar_deps_stale():
await asyncio.to_thread(_reinstall_sidecar_deps)
The patch spawn never got the same treatment. It is not startup-only
either — `_start_sidecar` is called from `connect()`, which takes
`is_reconnect`, so an ordinary Photon reconnect (network blip, sidecar
death) re-runs it and stalls a live gateway that is actively serving
Discord/Telegram/Slack traffic.
Dispatch it via `asyncio.to_thread` like its sibling. Same off-the-loop
class as the inbound-image decision (#66688) and the cron-fire verifier.
Adds a regression test asserting the spawn executes on a worker thread
rather than the loop thread.
`_reap_stale_sidecar` is `async`, but it identified the processes holding
the sidecar port with two blocking helpers called inline:
* `_find_listener_pids` -> `subprocess.run(["lsof", ...], timeout=5.0)`
* `_pid_is_sidecar` -> `subprocess.run(["ps", ...], timeout=5.0)`, once
per candidate pid
so the inspection can hold the shared gateway loop for 5 + 5·N seconds
while nothing else on it is serviced. It only runs once the /healthz probe
finds something already listening — the orphaned-sidecar recovery path —
and `_reap_stale_sidecar` is awaited from `_start_sidecar`, which runs on
every reconnect (`connect(is_reconnect=True)`). The stall therefore lands
on a live gateway that is still serving every other platform, right when a
crashed sidecar has already left an orphan behind.
Move the whole inspection to one `asyncio.to_thread` hop (one hop rather
than N+1 round trips). The reaping semantics are untouched: SIGTERM for
verified orphans, SIGKILL escalation, and both foreign-listener
RuntimeErrors behave exactly as before.
Same off-the-loop class as the inbound-image decision (#66688) and the
cron-fire verifier.
Adds a regression test asserting both the lsof lookup and the per-pid ps
check execute on a worker thread rather than the loop thread.
Fixes#73159.
When the Photon sidecar exits unexpectedly, _supervise_sidecar()
(running as self._sidecar_supervisor_task) correctly detects
SIDECAR_CRASHED and calls self._notify_fatal_error(). The Gateway's
fatal-error handler answers that by calling adapter.disconnect(),
which calls _stop_sidecar() -- from INSIDE the very task that's
currently executing this whole chain.
_stop_sidecar()'s cleanup unconditionally cancelled
self._sidecar_supervisor_task. Cancelling the currently-running task
raises CancelledError at its own next await point (inside
_notify_fatal_error() or _stop_sidecar() itself). Since
asyncio.CancelledError inherits from BaseException (not Exception),
the Gateway's `except Exception` guards around the fatal-error handler
don't catch it -- the handler aborts before ever reaching the
"queue platform for background reconnection" step. Photon then stays
permanently in `retrying` state until the whole process is manually
restarted, even though detection worked correctly and the underlying
transient upstream outage had long since recovered.
Fix: in _stop_sidecar()'s cleanup, check whether
self._sidecar_supervisor_task is asyncio.current_task() before
cancelling it. A task cannot legally cancel itself in any useful way
anyway (the cancellation only takes effect at its own next await,
which is exactly the corruption described above) -- when we're inside
the supervisor's own call stack, it's already in the process of
finishing on its own once _notify_fatal_error() returns, so skip the
cancel and just clear the reference.
This mirrors an existing precedent in the same file: disconnect()
already guards its OTHER task-cancellation (self._sidecar_health_task)
the same way (`if task is not asyncio.current_task()`), just not the
supervisor task in _stop_sidecar().
Per the issue's own note that existing tests mock out
_notify_fatal_error() entirely (so this integration chain was never
exercised), added two tests that drive the REAL chain: one runs
_supervise_sidecar() as an actual asyncio task with a real
_notify_fatal_error() that calls the real disconnect() -> _stop_sidecar(),
confirming the task completes without CancelledError and that the
post-disconnect reconnect-queue step actually executes; a second
confirms the OTHER call path (external cleanup, a different task)
still correctly cancels a running supervisor exactly as before.
Reverting only the adapter.py fix (keeping the new test) reproduces
the exact CancelledError from the bug report, confirming this is a
genuine regression test.
2 new tests pass; 113/113 in the full tests/plugins/platforms/photon/
directory (no regression to sidecar lifecycle, overflow recovery, or
health-monitoring behavior).
Follow-up to #69112. That PR hardened the shared gateway dispatch path in
gateway/run.py against the *caller* being cancelled. A second, self-referential
cancellation specific to PhotonAdapter sits one layer underneath it and survived
that fix.
PhotonAdapter is the only platform adapter that awaits _notify_fatal_error()
inline, on the same task that detected the fault. Both _monitor_sidecar_health
and _supervise_sidecar run as self._sidecar_health_task /
self._sidecar_supervisor_task, and the notification routes into
GatewayRunner._handle_adapter_fatal_error_impl, which tears the adapter down via
_safe_adapter_disconnect -> disconnect(). disconnect() then cancels
self._sidecar_health_task and awaits it -- which, when the health task is what
raised the notification, means disconnect() cancels its own caller several
plain-await frames up.
disconnect()'s `task is not asyncio.current_task()` guard does not catch this.
The current task where that guard evaluates is the wrapper
_await_adapter_cleanup_with_timeout creates around disconnect() via
asyncio.ensure_future, not the health task further up the chain, so the guard
passes and the cancel lands.
CancelledError stopped subclassing Exception in Python 3.8, so the
`except Exception` that wrapped the inline notify call never saw it. The health
task died silently mid-handoff: no log line, no "exception never retrieved"
warning (cancellation is normal asyncio), and no retry. The platform stayed
stranded until the gateway was restarted by hand.
Fix: dispatch the notification onto a new task, the same pattern
DiscordAdapter._handle_bot_task_done already uses for this reason. disconnect()
can then cancel the health/supervisor task freely without that cancellation
reaching the code still running the handoff, so the handoff always reaches the
reconnect queue. Both Photon fatal call sites are converted: the health-poll
path (observed wedging) and the sidecar-crash path (same shape, not yet
observed). gateway/run.py is untouched.
Observed twice on a self-hosted gateway, ~4h38m and ~52min of silent inbound
outage, both cleared only by a manual restart, both post-dating #69112's merge.
In each case the fatal log line appears and `queued for background reconnection`
never does.
Tests: new tests/plugins/platforms/photon/test_fatal_notify_self_cancel.py
covers the self-cancellation (fails with CancelledError without this change),
that the dispatch does not block its caller, that a failing notification warns
rather than raising, and a source guard against reintroducing either inline
await. Two assertions in test_overflow_recovery.py that drove these coroutines
directly now drain pending tasks before asserting delivery, since the
notification is deliberately no longer awaited inline.
Prepared with agent assistance and reviewed before submission.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two independent cross-PR collisions red on main (slice 8/8):
1. fd4f756492 (salvaged from stale #54514) added an early 'drop U+FFFC
placeholder' return at the top of _dispatch_inbound — written before
the deferred-wait handler (6b91b50c6e/afab7ed46e) existed further
down the same function. The early return shadowed it: _pending_fffc
never populated, no attachment-timeout tracking, 4 tests red. Remove
the duplicate block; the deferred handler already drops the
placeholder AND tracks/cancels/warns.
2. 9cf2046081 tightened sidecar_deps_installed() to require
node_modules/spectrum-ts, but test_runtime_record's _patch_spawn
fixture still created only bare node_modules/ — 2 tests red. Mirror
a real completed install.
tests/plugins/platforms/photon: 164/164 after; 158/164 before.
The sidecar auth token is generated at spawn (secrets.token_hex) and
existed only in the gateway process memory + sidecar child env, so
_standalone_send from cron subprocesses, hermes send, or the dashboard
structurally could not authenticate (#69960).
The adapter now writes <hermes-home>/runtime/photon-sidecar.json
({port, token, pid}, 0600, atomic tempfile+os.replace) once the sidecar
passes its /healthz readiness check, and deletes it in _stop_sidecar,
on every startup-failure path, and at disconnect so a stale record
never outlives a dead sidecar. _standalone_send falls back to the
record when PHOTON_SIDECAR_TOKEN is unset, validating the recorded pid
is alive first; a stale record yields a clear 'gateway appears to be
down' error. Docs note the gateway-must-be-running requirement and the
Photon-side shared-line initiation policy (#51897).
Maintainer follow-up: _cmd_setup now validates existing tokens (#72763
salvage); the pre-existing setup tests monkeypatch a stored token, so
without stubbing check_photon_token_valid they'd hit the real dashboard
API and hang.
Addresses teknium1 review on #50983:
- cli.py's `hermes photon status` and adapter.py's _start_sidecar() still
used the old node_modules/-existence check while check_requirements()
had moved to a spectrum-ts content check. Extracted the check into a
shared sidecar_deps_installed(), used by all three, so an
empty/partial node_modules/ (aborted npm install) is rejected
consistently instead of only in check_requirements().
- _install_sidecar()'s success-path _NPM_ERROR_LOG.unlink() only caught
FileNotFoundError, so a PermissionError/OSError on a locked file would
propagate. Broadened to OSError.
- npm stderr was truncated only when read back in check_requirements();
an unbounded stderr was written to disk on every failed install. Now
truncated to _NPM_ERROR_LOG_MAX_CHARS before write_text().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Problema
--------
Quando o npm install do sidecar Photon falhava, o Hermes descartava toda
evidencia e continuava normalmente - deixando o adapter de iMessage
silenciosamente ausente, sem nenhuma mensagem de erro acionavel.
Tres falhas independentes formavam o caminho de falha silenciosa:
1. check_requirements() sem logging
Cada branch de return False retornava sem emitir nenhum log. O core em
platform_registry.py so consome o bool de check_fn() e loga uma mensagem
generica com o install_hint - sem acesso ao motivo real da falha.
if not HTTPX_AVAILABLE: return False # sem log
if not shutil.which(node): return False # sem log
if not node_modules.exists: return False # sem log
2. node_modules/ parcialmente criado passava o guard (Risk 2)
npm cria node_modules/ antes de abortar em ENOSPC, timeout de rede ou
EACCES. O diretorio existia, check_requirements() retornava True (falso
positivo), o adapter era registrado, e o crash acontecia em runtime com
um erro de modulo ausente aparentemente nao relacionado ao setup.
3. stderr do npm descartado (Risk 3)
subprocess.run sem stderr=PIPE. O output de erro aparecia no terminal
durante o setup e sumia depois - diagnostico impossivel em CI/CD, Docker,
VPS headless, e qualquer reinstalacao posterior.
Correcoes
---------
adapter.py - check_requirements() agora loga por branch:
- httpx ausente -> logger.warning com nome do pacote
- node nao no PATH -> logger.warning com nome do binario e env var
- spectrum-ts ausente -> logger.debug com path do sidecar + ultimo erro npm
(DEBUG nao WARNING: estado normal pre-setup; check_fn() e chamado de
5 hot paths do core incluindo polling do /api/status)
adapter.py - content check em vez de existence check (Risk 2):
antes: if not (_SIDECAR_DIR / node_modules).exists()
depois: if not (_SIDECAR_DIR / node_modules / spectrum-ts).exists()
spectrum-ts e a unica dependencia do package.json. Checar sua presenca
garante que instalacao parcial/abortada e detectada no boot do gateway,
nao na primeira mensagem recebida via gRPC.
cli.py - stderr capturado e persistido (Risk 3):
subprocess.run passa agora stderr=subprocess.PIPE, text=True em ambas as
chamadas (npm ci e npm install fallback). O stderr capturado e:
- impresso em sys.stderr imediatamente (output visivel no terminal)
- persistido em _NPM_ERROR_LOG = sidecar/.photon-npm-error.log se
returncode != 0, limitado a 300 chars
- apagado de _NPM_ERROR_LOG se returncode == 0 (evita erro stale)
check_requirements() le _NPM_ERROR_LOG quando spectrum-ts esta ausente e
inclui o conteudo no DEBUG log - o erro do npm sobrevive ao terminal, ao
restart do gateway e a reinicializacao da maquina.
sidecar/.gitignore - adicionado node_modules/ e .photon-npm-error.log.
Isolamento - sem impacto no core:
- check_fn() continua retornando apenas bool; core nao e modificado
- Logging usa namespace plugins.platforms.photon.adapter, isolado de
gateway.* e hermes_cli.*
- Cada plugin tem seu proprio check_requirements() independente
- OSError no write/read de _NPM_ERROR_LOG e silenciado - nunca propaga
Testes - 24/24 passando:
test_check_requirements_risks.py (7 testes):
WARNING emitido quando httpx ausente
WARNING emitido quando node nao no PATH
DEBUG emitido (nao WARNING) quando spectrum-ts ausente, com path
node_modules/ vazio agora retorna False (Risk 2 resolvido)
_NPM_ERROR_LOG escrito no stderr do npm em falha
_NPM_ERROR_LOG apagado apos npm bem-sucedido
erro npm aparece no DEBUG log quando node_modules ausente
test_npm_error_log_regression.py (9 testes - vetores de falha da solucao):
return code contrato intacto (0 em sucesso, nao-zero em falha)
OSError no write do log silenciado, exit code ainda propagado
OSError no read do log silenciado, check_requirements() retorna False
stderr vazio nao cria arquivo de log
proc.stderr=None nao lanca AttributeError
log stale apagado apos reinstall bem-sucedido
DEBUG emitido mesmo sem log de erro (setup pela primeira vez)
Closes#50981
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hermes photon setup unconditionally called regenerate_project_secret()
on every re-run, invalidating the credential held by a running sidecar
and causing all outbound sends to fail with AuthenticationError.
Now validates existing credentials via a lightweight list_users call
before deciding to regenerate. Only rotates when no credentials exist
or the existing ones are invalid, and warns the user to restart the
gateway when rotation occurs.
Fixes#50755
store_photon_token/store_project_credentials/store_user_numbers read-modify-
wrote auth.json without hermes_cli/auth.py's _auth_store_lock(), the
cross-process flock every other writer of that file (credential_pool
refresh, model_switch, fallback_cmd, nous_portal adapter, etc.) already
holds. A concurrent write from either side during the unlocked
load-mutate-save window silently drops the other side's update.
Review follow-up: if os.fdopen() raised before taking ownership of the
descriptor returned by os.open(), the cleanup handler unlinked the temp
file but leaked the fd. Close it explicitly on that path, mirroring the
credential-writer cleanup from #62837.
Strengthen the tests so the old writer could not pass them: an os.open
spy asserts O_CREAT | O_EXCL and an explicit 0o600 mode (the final-mode
check alone was also satisfied by the post-write chmod), and a forced
fdopen-failure test asserts the raw fd is closed and no temp file is
left behind.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_save_auth() wrote the bearer token with tmp.open('w') — created at
process umask (typically 0o644) — and only chmod'ed to 0o600 after the
write, leaving a window where the token sat world-readable. The temp
name was also fixed and predictable (auth.json.tmp), so it could be
pre-planted (symlink attack).
Create the temp file with os.open(O_WRONLY|O_CREAT|O_EXCL, 0o600) and a
per-process random suffix, fsync before the atomic replace, and clean
the temp file up on failure. Mirrors hermes_cli/auth.py:_save_auth_store
(#19673, #21148), which hardened the same pattern in the core writer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Photon (iMessage) has no real edit API for already-sent messages. When
streaming completes, the gateway attempts to edit the message to remove
the streaming cursor (▉). Without edit support, this cursor gets stuck
in the final message, corrupting Unicode characters.
This change sets SUPPORTS_MESSAGE_EDITING=False on PhotonAdapter, which
causes the gateway to suppress the streaming cursor entirely for this
platform (via _effective_cursor in gateway/run.py). This prevents the
stale tofu square (▉) from appearing in streamed iMessage responses.
Fixes#49253
- Move U+FFFC placeholder detection before _record_last_inbound() so the
placeholder message id is not recorded as the reaction target
- Cancel pending U+FFFC tasks in disconnect() to prevent task leaks
- Check mimeType audio/x-caf in addition to filename for CAF→VOICE promotion,
fixing unnamed attachments that default to '(unnamed)'
- Add 7 Photon adapter tests: CAF named/unnamed promotion, U+FFFC no-dispatch,
U+FFFC not recorded as last inbound, U+FFFC+attachment cancel, timeout fire,
disconnect cleanup
- Add 6 transcription tests: ffmpeg conversion, afconvert fallback, all-fail,
CAF→WAV before Groq, conversion failure error, local provider skip
The native Gemini provider profile's default_aux_model and the curated
model picker catalog were still pinned to gemini-3.5-flash, a stale
generation now superseded by gemini-3.6-flash (documented GA). Bump
both so the auxiliary-task default and the picker stay in sync with
the current model.
Contract test asserts the durable lockstep invariant only
(default_aux_model is a member of _PROVIDER_MODELS["gemini"]) rather
than pinning either side to a frozen model-name string, so it doesn't
need updating on the next model-generation bump.
Stop offering deepseek-chat/reasoner in the static catalog and point
fallback/aux defaults at the permanent v4 IDs. Keep retired aliases in
a detection-only map so /model deepseek-chat still resolves to deepseek.
The sidebar strip and the Channels page could contradict each other on
the same page load — "Gateway running" next to "The gateway is not
running." /api/status and /api/messaging/platforms each open-coded their
own liveness ladder: status probed GATEWAY_HEALTH_URL and scoped its
PID/state reads to the requested profile, messaging did neither and used
the uncached raw PID probe.
Three deployments hit the split: a cross-container gateway (no local PID,
only the health probe can see it), a profile-scoped dashboard (messaging
borrowed a DIFFERENT profile's runtime state, reporting a false
"connected" that hides a real outage — #71211), and a launch-service
managed gateway with no PID file.
Adds resolve_gateway_liveness() in gateway/status.py as the single ladder
(cached PID -> HTTP health probe -> runtime-status PID with
expected_home) and routes both endpoints, /api/messaging/platforms/{id}/test,
and the kanban dispatcher-presence probe through it. Probe callables are
injectable so the existing monkeypatch seams keep working, and
GatewayLiveness.probe_error distinguishes "down" from "couldn't tell" so
the kanban warning keeps failing OPEN instead of crying wolf.
Closes#71211.
The Codex image backend rejected our own request shape for every account, and
we then translated that rejection into "Image generation is not enabled for the
current Codex account. Switch the image provider to OpenAI API key, FAL, or
xAI." — telling every affected user to abandon a provider that had never
actually been tried. That message is why this reads as a setup failure rather
than a bug: the wire error was replaced with a confident, wrong diagnosis.
Removes the classifier and its exception, so any HTTP failure surfaces
verbatim. The paired request-shape fix (previous commit) is what makes the
400 stop happening; this commit makes the next one diagnosable.
Also fixes error-body truncation: bodies were head-truncated at 500 chars, and
Codex error payloads can carry hundreds of bytes of leading metadata, so the
user got a wall of padding and no message. _summarize_error_body() prefers the
parsed error.message and falls back to a truncated raw body.
Docs: drop the unqualified image-to-image claim for the Codex backend and note
that the hosted tool call cannot be forced, so it is best-effort.
Verified E2E against a local fake Codex backend: success path writes a real PNG
with no tool_choice on the wire; the 400 path now returns api_error carrying
"Tool choice 'image_generation' not found in 'tools' parameter" (148 chars)
instead of the entitlement message. Sabotage run confirms all 4 regression
tests fail when the old behavior is restored.
Refs #19505, #49008, #31335.
The chatgpt.com/backend-api/codex backend 400s on every tool_choice
shape for the hosted image_generation tool — it looks up tool_choice as
a function name and never recognizes hosted-tool entries. Removing the
field from _build_responses_payload() lets the host model decide; the
instructions field nudges it toward the tool.
Salvaged from PR #19979 (originally targeted the old
client.responses.stream call, which no longer exists on upstream/main;
the live request now flows through _build_responses_payload + httpx in
_collect_image_b64).
Follow-up to review feedback:
- mem0 _prompt_api_key read .env with the locale default, so a Notepad
BOM hid the first key from the masked current-value lookup; read it
with utf-8-sig + errors=replace like the canonical readers in
hermes_cli/config.py.
- hindsight _load_simple_env used plain utf-8; it also parses the Hermes
.env during post_setup, where a BOM stuck to the first key. Switch to
utf-8-sig + errors=replace.
- Add hindsight regressions: BOM key matching in _load_simple_env and in
the cloud post_setup writer, plus non-ASCII round-trip preservation,
and a mem0 regression for the BOM'd masked-key lookup. The BOM tests
fail without the fix on any platform.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mem0 and hindsight memory-provider setup routines round-trip the
user's ~/.hermes/.env: they read existing lines, update the keys they
manage, and rewrite the whole file preserving every other line verbatim.
Both used env_path.read_text() / write_text() with no encoding.
read_text()/write_text() with no encoding fall back to the system locale
(cp1252/GBK on Windows), so on a non-UTF-8 host the preserved lines get
mangled or the call crashes on any non-ASCII value, and — because the
reader never strips a BOM — a Notepad-edited .env makes the first key
fail the in-place match and get duplicated instead of updated.
Match the canonical .env readers in hermes_cli/config.py: read with
encoding='utf-8-sig' (BOM-tolerant) and write with encoding='utf-8'.
mem0/_setup.py already pins utf-8 for mem0.json, so this just aligns the
.env path in the same file. Fixes both memory plugins in one class fix.
Adds regression tests: a BOM'd .env updates the first key in place
(locale-independent, fails without the fix) and non-ASCII existing lines
survive the round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Avoid spawning multiple local OpenViking server processes while a runtime autostart waiter is already active. Remote endpoints still retry on later accesses because they do not install a local waiter.
The _needs_trusted_identity_retry method was hard-coding specific
server-side error strings to detect when a request failed due to
missing X-OpenViking-Account / X-OpenViking-User headers. Each new
server-side error variant required another string added to the client.
Replace the string enumeration with a structural match: the error
message mentions one of the tenant headers AND the HTTP status is 400.
This covers all current error variants:
- "Trusted mode requests must include X-OpenViking-Account and User"
- "ROOT requests to tenant-scoped APIs must include X-OpenViking-Account"
- "Trusted mode requests must include X-OpenViking-Account."
- "Trusted mode requests must include X-OpenViking-User."
The 400 status guard avoids false-positives on 403 errors such as
"USER API keys cannot override X-OpenViking-User", which must not
trigger a retry.
All 176 existing tests pass.
(cherry picked from commit 5a24d6766c)
`OpenVikingMemoryProvider.shutdown()` joins in-flight writers, deferred-commit
threads, and prefetch threads, but not `_runtime_start_thread` — the tracked
`daemon=True` waiter that runs `_finish_runtime_openviking_start`, which blocks
on network health probes (`_wait_for_openviking_health` polling + a
`_VikingClient.health()` request).
If the local OpenViking runtime is slow or unreachable, that waiter can still
be blocked in network I/O at interpreter exit. CPython then forcibly kills it
during `Py_FinalizeEx` (`PyThread_exit_thread` -> `__pthread_unwind` ->
`abort()`), producing SIGABRT (exit 134) with no traceback — the same daemon-
thread-at-exit failure class fixed for the Honcho provider.
Fix:
- `shutdown()` now joins `_runtime_start_thread` (timeout-bounded) alongside the
other tracked threads.
- `_wait_for_openviking_health()` gains a `should_stop` callback; the waiter
passes `lambda: self._shutting_down` so the poll loop bails out promptly once
`shutdown()` flips the flag, instead of lingering up to the 60s autostart
timeout and timing out the join (which would leave the thread alive).
- Add tests/plugins/memory/test_openviking_shutdown.py covering the short-circuit
and the shutdown-joins-runtime-thread behaviour.
(cherry picked from commit 5471ec70210b35462450aa52f0cd483439fffba7)
`_write_env_vars` in the OpenViking memory provider interpolates each
secret straight into a `KEY=VALUE` line, but the values only ever pass
through `_clean_config_value`, whose `value.strip()` trims surrounding
whitespace and leaves internal CR/LF intact. Because the file is strictly
line-oriented and is re-read via `read_text().splitlines()`, a value that
carries an embedded newline spills onto a second physical line, and the
tail is re-parsed as an independent `KEY=VALUE` entry on the next round
trip. A secret pasted with a trailing record (e.g. an `OPENVIKING_API_KEY`
copied with an extra line) therefore injects an arbitrary additional
variable into the persisted credentials file and silently corrupts it.
The fix neutralizes the line terminators at the single chokepoint where
values reach the file. A small `_env_line_safe` helper strips `\r`, `\n`,
and the NUL byte from each value, and both write sites in `_write_env_vars`
(the existing-key update branch and the appended-key branch) route through
it, so a value can only ever occupy the single line it is written on.
## What does this PR do?
Hardens the OpenViking memory provider's `.env` writer so a malformed or
pasted secret value can no longer break out of its `KEY=VALUE` line and
inject a rogue variable into the profile-scoped credentials file.
## Related Issue
N/A
## Type of Change
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `plugins/memory/openviking/__init__.py`: add `_env_line_safe()` which
removes `\r`, `\n`, and `\x00` from a value, and apply it to both the
updated-key and appended-key write branches in `_write_env_vars()`.
- `tests/plugins/memory/test_openviking_provider.py`: add two regression
tests covering a fresh write and an in-place key update with embedded
CR/LF, asserting no injected line survives the read-back.
## How to Test
1. Run the targeted tests:
`pytest tests/plugins/memory/test_openviking_provider.py -k env_writer -q`
2. Reverting the `_env_line_safe` sanitization makes
`test_openviking_env_writer_strips_embedded_newlines_in_values` and
`test_openviking_env_writer_strips_newlines_when_updating_existing_key`
fail with a rogue `INJECTED_KEY=`/`ROGUE=1` line appearing in the file,
confirming the tests pin the bug.
3. `ruff check plugins/memory/openviking/__init__.py` and
`python scripts/check-windows-footguns.py plugins/memory/openviking/__init__.py`
both pass.
## Checklist
### Code
- [x] I've read the Contributing Guide
- [x] My commit messages follow Conventional Commits
- [x] I searched for existing PRs to make sure this isn't a duplicate
- [x] My PR contains only changes related to this fix
- [x] I've run the relevant tests and they pass
- [x] I've added tests for my changes (required for bug fixes)
- [x] I've tested on my platform: macOS 15 (Darwin 25.5)
### Documentation & Housekeeping
- [x] I've updated relevant documentation (docstrings) — or N/A
- [x] I've updated `cli-config.yaml.example` if I added/changed config keys — N/A
- [x] I've updated `CONTRIBUTING.md` or `AGENTS.md` if I changed architecture or workflows — N/A
- [x] I've considered cross-platform impact (strips CR as well as LF) — done
- [x] I've updated tool descriptions/schemas if I changed tool behavior — N/A
(cherry picked from commit f29dd2df84)
