Teardown-path tests build bare runners via object.__new__ without
the liveness-guard machinery; the unguarded call raised
AttributeError in 8 tests. Same guard pattern as the start path.
Follow-up to the salvaged #69164 commits: policy forbids introducing new
HERMES_* environment variables, so the four watchdog env knobs
(HERMES_GATEWAY_LOOP_WATCHDOG / _INTERVAL / _TIMEOUT / _STRIKES) are
replaced with a single config.yaml boolean:
gateway:
loop_watchdog: true # default; false disables both guards
- gateway/config.py: new GatewayConfig.loop_watchdog field (default True),
parsed from top-level or nested gateway: form, round-trips via
to_dict/from_dict.
- gateway/run.py: _start_loop_liveness_guards() checks config.loop_watchdog
before arming the floor timer + watchdog (getattr-guarded for bare
object.__new__ runners).
- gateway/shutdown_watchdog.py: start_loop_liveness_watchdog() no longer
reads the environment; probe interval/timeout/strikes are module
constants (30s/10s/3 — ~90s to restart, matching the systemd watchdog
layer's posture).
- hermes_cli/config.py: documented gateway.loop_watchdog default so
'hermes config set gateway.loop_watchdog false' validates.
- tests: env-knob tests replaced with config-gate + round-trip tests;
the final-strike boundary test injects its probe via max_strikes
directly instead of patching the removed env helper.
- A stop() landing while the final diagnostics (critical log,
traceback dump) are executing could still reach os._exit(75) after
the pre-diagnostic check. Add a third stop_event recheck immediately
before the hard exit: diagnostics may complete, but a disarmed
watchdog never exits.
- Deterministic regressions for both windows (stop triggered from
inside logger.critical and from inside faulthandler.dump_traceback);
mutation-verified (removing the check turns both red). Frozen-loop
semantics unchanged.
Addresses the second round of the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Re-check stop_event after a missed probe (before the strike
increment) and again on entering the final-strike branch (before the
critical log, dump, and hard exit), so a normal stop() landing
between the last timeout check and the exit path can no longer be
misclassified as a freeze and trigger a supervisor restart.
- Deterministic boundary tests pin both re-checks independently
(mutation-verified: removing either check turns its own test red);
frozen-loop semantics are unchanged.
Addresses the shutdown-race review on #69164.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A self-rescheduling 5s call_later floor timer, armed before any
adapter connects, guarantees the selector always has a finite
timeout, so the existing async defenses (polling heartbeat, timeout
guards) regain a chance to run after a zero-pending-timer stall.
- A resident daemon-thread liveness watchdog probes the loop via
call_soon_threadsafe every 30s; after 3 consecutive 10s-timeout
misses (~120s of total unresponsiveness) it dumps all thread
tracebacks and exits with the established
GATEWAY_SERVICE_RESTART_EXIT_CODE (75) so a supervisor restarts the
gateway - async-level recovery cannot run on a frozen loop.
- stop() disarms both guards before any teardown await so a busy
shutdown is never misjudged as a freeze.
HERMES_GATEWAY_LOOP_WATCHDOG=0 disables; _INTERVAL/_TIMEOUT/_STRIKES
tune the thresholds.
Fixes#69089
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
signal.SIGUSR2 and faulthandler.register() don't exist on Windows;
the bare reference raised AttributeError at import time per the
windows-footgun checker. faulthandler.enable() still covers
fatal-error dumps on all platforms.
When connected_count == 0 and at least one platform failed with a
non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE
(78) even if OTHER platforms failed for merely transient reasons.
Real-world shape (NS-609, hosted instance): WhatsApp enabled but never
paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during
polling startup (retryable) => exit 78 => the gateway either goes
permanently down (supervisors honoring the exit-78 contract via
RestartPreventExitStatus / the s6 finish->125 translation from #51228) or
crash-loops (anything else). Either way Telegram never gets its retry and
the dashboard drops with every exit, so a single unpaired platform plus
one network blip disconnected every channel on the instance.
Now exit 78 is reserved for the case where ALL startup failures are
non-retryable (true config error, nothing to wait for). With mixed
failures the gateway stays alive in degraded state: the reconnect watcher
recovers the retryable platforms and the misconfigured ones stay
fatal-parked and visible in runtime status.
Three-part fix for the gateway going silently deaf after a retryable
fatal adapter error (e.g. httpx.ConnectError on Telegram):
1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced
plain asyncio.wait_for with the task-detach pattern used by
_await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the
overdue task but then waits for it to exit, so a connect() that
catches CancelledError can block recovery forever. The detach
pattern releases the runner at the deadline via
consume_detached_task_result.
2. **Ensure reconnect watcher always runs after escalation** — Added
_ensure_reconnect_watcher_running(), called after queueing a
retryable fatal error. If the reconnect watcher task has died
(exhausted restart budget, terminal exception), it is respawned
so queued platforms are never permanently stranded.
3. **Faulthandler at gateway startup** — Enabled faulthandler +
SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for
post-mortem diagnosis of future event-loop freezes.
Tests added for _ensure_reconnect_watcher_running (alive, dead,
not-started, not-running), fatal-error integration (retryable calls
ensure, non-retryable does not), and _connect_adapter_with_timeout
(timeout raises, success returns).
Builds on this PR's diagnosis by @Frowtek: a missing workdir is
ambiguous (deleted project vs. an unmounted external volume / network
share / VPN not yet up), so it's not safe evidence for a destructive
GC sweep — especially one that runs unattended at startup.
- cli.py / gateway/run.py: the startup auto-maintenance sweep now
always passes delete_orphans=False to maybe_auto_prune_checkpoints().
It still prunes by retention_days, size cap, and legacy archives —
none of which require guessing whether a project was deleted or is
just temporarily unreachable.
- hermes_cli/config.py: drop the now-unused delete_orphans default.
- hermes_cli/checkpoints.py: `hermes checkpoints prune` (the explicit,
human-invoked path) now previews the orphan project list and asks
for confirmation before deleting, unless -f/--force is passed.
- Docs updated (EN + zh-Hans) to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sibling of the #69678/#69567 ledger leak class found while widening the
sweep: _probe_state_db used 'with sqlite3.connect(...)', whose context
manager only commits/rolls back and never closes, leaking one connection
(db fd) per health poll in the long-running gateway. Wrap the connection
in contextlib.closing so every probe closes deterministically.
Three durable ledgers used `with _connect() as conn:` where the sqlite3
connection context manager commits/rolls back but never closes, leaking the
db/-wal/-shm file descriptors on every call. On a long-running gateway this
exhausts RLIMIT_NOFILE and fails unrelated components with
`[Errno 24] Too many open files`. Same bug class as the cron execution ledger
(#69567 / PR #69594), which the connection helpers here are modeled on.
Fix: route every ledger operation through a `_transaction()` context manager
that guarantees `conn.close()` on exit. `_connect()` keeps its
schema-on-connect contract (several tests call it directly) and now self-closes
if schema init fails.
Adds per-module regression tests asserting every opened connection is closed,
including the no-op-update and exception-mid-transaction paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a persisted, backend-confirmed provider/model lock for Hermes
Browser and other session API clients. A confirmed lock is an
execution contract rather than response metadata:
- POST /api/sessions/{session_id}/model validates and persists a
confirmed browser_model_lock (advertised in /v1/capabilities)
- session chat + chat/stream consume the persisted lock on body-only
follow-up turns; a confirmed lock wins over an older gateway session
/model override and the session-persisted model
- a later successful session /model switch explicitly clears and
replaces the lock while preserving lineage markers (_branched_from)
and invalidating cached system-prompt model/provider metadata
- ordinary one-off request overrides never replace a confirmed lock
- provider-resolution failure fails closed as a typed provider-auth
error (controlled response, never global-credential reuse)
- confirmed locks disable the global fallback model chain
- the completed agent's actual provider/model must match the locked
route or the turn fails with a runtime-mismatch error
- responses carry sanitized runtime metadata reporting actual vs
requested provider/model and lock state
Rebased onto the provider-aware request routing (#70853) and
session-model parity (#70931) that landed since the original branch;
the lock now slots into that precedence chain as the top rung.
Salvaged from PR #61236 by @abundantbeing.
Three parity fixes between the API server and the native gateway's
agent-runtime resolution, integrated with the provider-aware request
routing that landed in #70853:
- Session-persisted model is honored: POST /api/sessions {"model": ...}
stores a model that the chat handlers previously fetched and threw
away. A stored value that matches a model_routes alias goes through
the route path (route provider/credentials apply); a raw model string
threads through as session_model, pinning the session's turns ahead
of per-request body values but below an explicit session /model
override.
- Empty-model recovery: provider-catalog default when config has no
model.default but a provider resolved, plus last-known-good model
recovery (#35314) keyed on gateway_session_key only (never ephemeral
session_id — no unbounded growth from one-off requests).
- Provider auth failures surface as controlled responses: RuntimeError
from _resolve_runtime_agent_kwargs() is re-raised as a dedicated
_ProviderAuthResolutionError at the call site, caught narrowly in
_run_agent() and the /v1/runs executor to return run.py's response
shape instead of an undifferentiated 500 (session-chat endpoints
previously returned a raw aiohttp 500 with no JSON body).
Salvaged from PR #57947 by @FvanW; session-model route-alias resolution
from PR #59941 by @kaishi00.
Co-authored-by: kaishi00 <kaishi00@users.noreply.github.com>
AST-driven pass over every subprocess.run/Popen/check_output/check_call/call
with text=True (or universal_newlines=True) and no explicit encoding=:
append encoding='utf-8', errors='replace' at the kwarg site. 136 call
sites across 28 files (cli.py, hermes_cli/main.py, tools_config.py,
environments, computer_use, gateway, scripts, skills helpers, agent/*).
Together with the salvaged #55339/#60741 commits this closes out issue
#53428's bug class; the salvaged #60751 linter rule in
check-windows-footguns.py now enforces it repo-wide (verified: 807 files
scanned, zero findings).
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
Add authenticated GET /api/model/options to the gateway API server,
sharing the dashboard/TUI picker payload builder so external clients
can sync to the user's configured Hermes provider catalog instead of
scraping the single OpenAI-compatible /v1/models alias.
- new shared hermes_cli.inventory.build_model_options_payload() wraps
build_models_payload with the stable picker shape and safe
custom-provider probe policy (probe current only on normal open,
probe all + cache bust on explicit refresh)
- dashboard web_server and TUI gateway model.options refactored onto
the shared builder; dashboard build moved off the event loop via
run_in_threadpool
- capabilities endpoint advertises model_options
- docs for both API server and programmatic integration
Salvaged from PR #54689 by @abundantbeing.
Follow-ups on the salvaged #54426 routing contract:
- Bare `model` without `provider` on the OpenAI-compatible endpoints
(/v1/chat/completions, /v1/responses) is now opt-in via
gateway.platforms.api_server.direct_model_requests (default off) —
generic OpenAI clients hardcode model names ('gpt-4o', ...) and
existing deployments rely on those falling back to the gateway
default. Explicit `provider` requests and the Hermes-native
session-chat + /v1/runs surfaces are always honored.
Idea credit: PR #22825 by @mssteuer.
- A model_routes alias with no `model` key can no longer leak the
alias string as the executing model name (defensive; parse-time
validation already drops such routes).
- Fix mis-indented _run_agent call args in _handle_session_chat_stream.
- Docs: document the opt-in flag.
Carry model, provider, and model_options through the API server's
execution surfaces (session chat, Chat Completions, Responses, /v1/runs)
without mutating global configuration. Precedence: session /model
override -> model_routes alias -> direct request selection -> global
defaults. Conflicting route/provider mixes fail closed with 400.
model_options stays request-scoped regardless of which selection wins.
Salvaged from PR #54426 by @abundantbeing.
New sessions.auto_archive / auto_archive_days config: soft-hide (never
delete) sessions with no activity for N days, aging on last activity
rather than creation so an old-but-active chat is spared. Sweeps are
throttled through state_meta and fire from CLI startup, gateway startup
+ hourly housekeeping, and the serve/dashboard backend (opportunistic
on session list + an hourly lifespan ticker), so every surface honours
one setting.
A new pinned column (declaratively migrated) exempts sessions from the
sweep; PATCH /api/sessions/{id} accepts pinned and flips the whole
compression lineage as a unit, mirroring set_session_archived.
Under multiplex_profiles, the gateway starts a single InProcessCronScheduler
bound to the process-global HERMES_HOME (the default profile's home), so
only that profile's cron/jobs.json is ticked. A job registered from a
secondary-profile session lands in <profile>/cron/jobs.json, reports a valid
next_run_at — and never fires.
Changes:
1. cron/scheduler_provider.py — InProcessCronScheduler.start() now accepts
an optional profile_homes kwarg (list of (name, Path) tuples). When set,
_start_multiplex() iterates tick() over each profile home using
use_cron_store(), so every served profile's cron store is ticked on
every tick cycle. Heartbeats and interrupted-execution recovery are also
scoped per profile via use_cron_store().
2. gateway/run.py — start_gateway() now resolves profiles_to_serve(multiplex=True)
when multiplex_profiles is on and passes them to the cron scheduler as
profile_homes. Only applies to InProcessCronScheduler (the built-in);
external providers are unchanged.
3. cron/jobs.py — record_ticker_heartbeat(), get_ticker_heartbeat_age(), and
get_ticker_success_age() now resolve paths via _current_cron_store()
instead of module-level TICKER_HEARTBEAT_FILE / TICKER_SUCCESS_FILE
constants. This makes heartbeats correctly scoped per profile, so
'hermes cron status' reflects liveness for every profile independently
under multiplex_profiles.
4. tests/cron/test_scheduler_provider.py — two new tests:
- test_multiplex_ticker_ticks_each_profile_once: verifies tick() is called
once per profile per tick cycle.
- test_multiplex_heartbeat_scoped_per_profile: verifies heartbeat files
are written to each profile's cron store.
Routine automatic compression stays silent-by-design on chat platforms
(default unchanged, byte-identical). New opt-in config key
compression.progress_notices (bool, default false) opens a gate on the
gateway noise filter (_prepare_gateway_status_message) that lets ROUTINE
compression progress statuses through to chat surfaces.
- Membership is derived from the #69550 status template constants in
agent/conversation_compression.py (compiled to a literal-escaped regex,
never re-inlined wording), so unrelated noisy statuses (aux failures,
provider retry/rate-limit chatter) stay suppressed even when enabled.
- The compaction completion notice (COMPACTION_DONE_STATUS, #69546
lifecycle 'compacted' edge) already flows through the status path and
passes the filter — no new emit site needed.
- Config wired everywhere: hermes_cli/config.py DEFAULT_CONFIG,
cli-config.yaml.example, gateway raw-YAML read (live, mtime-cached),
gateway hot-reload cache-busting key list, website configuration docs.
- Failure notices and manual /compress feedback remain always-visible;
VISIBLE_COMPRESSION_MESSAGES and emit sites untouched.
Design by @havok-training (issue #52995).
The Windows /restart watcher's outer Popen spawns the watcher with
windows_detach_popen_kwargs() (which carries CREATE_BREAKAWAY_FROM_JOB),
but a restrictive parent job object can reject that bit with OSError and
the current call has no retry. Preserve the current watcher
implementation and add a focused breakaway-denied fallback.
Preserved from current main: watcher_python / pythonw.exe selection, the
str(restart_after_s) deadline, the scrubbed watcher_env, the intentional
no-breakaway inline respawn, and the entire POSIX setsid/bash path.
- primary keeps **windows_detach_popen_kwargs()
- on OSError, retry the same argv/env with
creationflags=windows_detach_flags_without_breakaway()
- on dual failure, log a definitive, path-safe warning (interpreter
basename + numeric winerror/errno only) and return without crashing
Replace the superseded breakaway-first inline design and its AST tests
with focused behavioral coverage that drives the real coroutine with a
mocked subprocess.Popen (retry, argv/env/DEVNULL preservation, POSIX
single-session kwarg, no-breakaway inline respawn, secret-safe logging).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add _ensure_last_n_user_messages_in_tail to guarantee the last N user
messages survive compression in the uncompressed tail, with surrounding
assistant/tool context preserved.
- Add min_tail_user_messages parameter (default 3) to ContextCompressor
- New _ensure_last_n_user_messages_in_tail method generalizes single-user protection
- Skip context-summary handoff banners when counting user messages
- User messages are clean boundaries — skip _align_boundary_backward
- Wire through cli.py, agent_init.py, and gateway cache busting keys
Config:
compression:
min_tail_user_messages: 3
Co-Authored-By: Claude <noreply@anthropic.com>
Follow-ups on top of the cherry-picked #62644 mechanism, porting it to
current main and closing the salvage-review requirements:
- proactive_prune_min_reclaim_tokens (default 4096): a prune only COMMITS
when it reclaims a meaningful token batch, measured on the pruned output.
A committed prune rewrites already-sent history and invalidates the
provider prompt-cache prefix; this hysteresis gate keeps those breaks
episodic/amortized (like a compression boundary) instead of firing every
tool iteration. 0 disables the gate. (Design point credited to the
#62389 review cycle's prune_minimum_tokens.)
- Standard no-op caller contract: every skip path returns the INPUT list
object; the loop commits only on 'result is not messages' + non-zero count.
- Loop call is getattr+callable guarded (plugin engines predating the hook,
SimpleNamespace test doubles) and exception-swallowed at debug level.
- Config parse follows the compression.max_attempts hardened semantics:
booleans rejected, fractional floats rejected, integral floats/numeric
strings accepted; negative trigger = disabled.
- cli-config.yaml.example documented (all three keys) and gateway
_CACHE_BUSTING_CONFIG_KEYS extended so hot-reload rebuilds the agent.
- Tests: min-reclaim gate both directions, input-object no-op contract,
no-orphan tool_call_id pairing in BOTH directions (#69830 pin rule),
default-off zero-behavior-change pin, config parse seam, and behavioral
loop-wiring tests (consulted/commit/no-op/absent-method/raising).
Gateway half of the #49874 salvage: pass compression_deferred through
both _run_agent_inner result dicts and guard the compression-exhausted
auto-reset block with it — a lock-contended defer keeps the session
intact (the concurrent compressor is actively shrinking it) instead of
wiping it via reset_session.
Regression tests:
- tests/run_agent/test_compression_lock_defer.py — provider-mock 413 and
400-overflow turns whose compression pass lost the lock end as
compression_deferred (failed=False, no compression_exhausted); flag
unset keeps the terminal exhaustion path byte-identical; type-pin
tests vs MagicMock agents and junk flag values; cap=1 e2e proving the
refunded pre-API defer leaves the budget for the provider-proven
413 retry.
- tests/agent/test_preflight_lock_defer.py — a lock-skipped preflight
pass stops the loop WITHOUT arming preflight_compression_blocked;
plain no-op still arms it; MagicMock junk does not defer.
- tests/gateway/test_compression_deferred_soft_result.py — AST pin that
the deferred branch guards the auto-reset chain and performs no
session mutation (mirrors test_35809_auto_reset_clean_context.py).
Gateway tests construct GatewayRunner via object.__new__ without __init__;
the new #51899 guard accessed self.config directly and crashed those
runners (AttributeError). Use getattr with None default — the guard
no-ops without config, matching the documented object.__new__ pitfall.
Previously, Slack sessions and the channel directory stored raw channel
IDs (e.g. C0ATFHY907L) as chat_name, making it impossible for operators
to identify channels in send_message listings or session data.
Changes:
- Add _channel_name_cache and _resolve_channel_name() to SlackAdapter
that calls conversations.info (channels) or users.info (DMs) with
in-memory caching to avoid repeated API calls
- Use resolved names in _handle_slack_message() build_source() calls
- Use cached names in _seed_assistant_thread_session()
- Enrich session-sourced entries in channel_directory._build_slack()
by cross-referencing API results and falling back to conversations.info
+ users.info for DMs and private channels not in the bot's scope
Fixes: channel directory and session origins now show readable names
like 'general' or 'John Doe' instead of 'C0xxx' / 'D0xxx'.
Follow-up to the #51899 pick, folding in the config-bridge half of the
competing PR #46925 (@bhanusharma, earliest submitter for the
ignored-channel gate):
- _apply_yaml_config: translate config.yaml slack.ignored_channels into
SLACK_IGNORED_CHANNELS (list or CSV), env-var-wins like every other
bridged Slack key.
- SlackAdapter._slack_ignored_channels / gateway.run's
_slack_ignored_channels_from_gateway_config: fall back to the
SLACK_IGNORED_CHANNELS env var when PlatformConfig.extra carries no
value, so top-level slack: blocks (which flow through the env bridge,
not extra) are honored at both the adapter and runner gates.
- conftest: force-clear SLACK_ALLOWED_CHANNELS / SLACK_IGNORED_CHANNELS /
SLACK_DISABLE_DMS between tests (config-loader side-effect leak class).
- Tests: env-bridge translation + precedence in test_config.py, env
fallback + extra-wins in test_slack_runner_ignored_channels.py.
Credit: #46925 by @bhanusharma proposed the same gate with the YAML→env
bridge; #51899 (picked as the base) carries the wider outbound/runner
coverage. Closes#46925 as consolidated here with first-submitter credit.
.get("key", {}) only applies the default when the key is ABSENT.
When the key exists with value None (null in JSON), .get() returns
None and the subsequent .get() raises AttributeError.
Fix: replace .get("key", {}).get(...) with (.get("key") or {}).get(...)
which handles both missing keys AND None values.
8 instances across 6 files:
- gateway/run.py: tool_call function name check
- acp_adapter/server.py: tool name/description extraction
- gateway/platforms/qqbot/onboard.py: API response task_id
- gateway/platforms/yuanbao.py: message content parsing (x2)
- gateway/platforms/slack.py: block text extraction (x2)
- tui_gateway/server.py: error message extraction
Conflict-resolution follow-up composing #68925 (Bob) with the already-
applied #20583/#66398 (jordanhubbard) recovery design:
- #68925's caller-level second _query_recoverable_session pass (via
lookup_session_key=) referenced a variable that no longer exists —
the legacy exact-key fallback now lives INSIDE
_query_recoverable_session, which also claims the legacy key once per
process and rewrites the peer row to the scoped key. Drop the dead
caller-level pass.
- Keep #68925's _recovered_row_matches_source_scope origin guard wired
into both recovery paths: a scoped channel lookup refuses rows whose
recorded origin names another workspace (or no workspace at all).
- Routing-index migration adoption policy documented at the site:
origin names a workspace -> exact match only; scope-less DM -> first
workspace claims once; scope-less channel -> refuse.
/v1/runs bound only session_key at its _bind_api_server_session call, so
tools.async_delegation._current_origin_session_id() — which reads the
request-scoped HERMES_SESSION_CHAT_ID — returned "" on that route and
runs-originated background delegations stayed forced-sync with no wake
target. Bind chat_id/session_id the same way the other agent-entry routes
do via _run_agent(). Follow-up to #64998 (sweeper review F2).
On non-push adapters (api_server) the wake self-post IS the delivery, but
the cursor advanced before the self-post ran and a failed/exhausted post
was swallowed by the best-effort except — permanently losing the event.
Reorder the else-branch: for non-push adapters run the self-post FIRST and
only advance the cursor once it succeeds. A failure rewinds the pre-send
claim (same guarantee as the existing SendResult(success=False) path) so
the next tick retries, with the same MAX_SEND_FAILURES drop threshold.
Push-capable adapters keep the pre-existing advance-then-best-effort-wake
behavior. Follow-up to #64998 (sweeper review F1).
Wake-ups for kanban notifications and background delegation completions were
injected via handle_message() using a build_session_key()-derived key, which
can never match the raw X-Hermes-Session-Id key that api_server sessions run
under — so the wake landed in a session nobody was reading. On top of that,
ApiServerAdapter.send() reports failure without raising, and that was treated
as a successful delivery, so the notify cursor advanced past events that were
permanently lost; and background delegation was forced synchronous on
api_server since there was no way to wake the session afterward.
Fix: route wake-ups for non-push adapters through a self-post to
/v1/chat/completions with the original session id, treat non-raising send
failures as failures (rewind instead of advancing the cursor), and re-enable
background delegation whenever a session id is available to wake.
The origin session id is captured from the request-scoped api_server chat_id
binding rather than HERMES_SESSION_ID: constructing a child agent calls
set_current_session_id() with the subagent's internal id, clobbering that
variable right before dispatch would read it and misrouting the wake into
the subagent's own session.
Related: #56580, #64609, #53027, #63169, #56531, #50319, #64113
The api_server adapter returned error code "invalid_api_key" for
API_SERVER_KEY authentication failures, which the Desktop error
classifier misidentified as a provider (OpenRouter/OpenAI) key
problem — showing "OpenRouter API key missing" when the real issue
was gateway auth.
Changes:
- gateway/platforms/api_server.py: return "gateway_auth_failed" code
with descriptive message for API_SERVER_KEY auth failures
- apps/desktop/src/store/notifications.ts: add "gateway_auth_failed"
handler before "invalid_api_key" to show correct error message
- agent/error_classifier.py: add "gateway_auth_failed" to auth patterns
- tests: update test_session_api.py to expect new error code
Fixes#39365
The gateway ignored config when defined through
YAML (config.yaml). The API server only started when environment
variables like and were set,
even though other gateway subsections like were
processed correctly.
Two changes in gateway/config.py's load_gateway_config():
1. Merge platform configs placed directly under gateway.*
(e.g. gateway.api_server) via _merge_platform_map, matching
the existing gateway.platforms.* and platforms.* merge
paths. Only keys matching known Platform values are picked up;
non-platform keys like streaming are safely ignored.
2. Bridge api_server-specific keys (port, key, host, cors_origins,
model_name) from the top-level config block into the extra
dict so PlatformConfig.from_dict preserves them — matching what
_apply_env_overrides already does for env var values.
When _api_key_passes_startup_guard() rejects the key (missing,
placeholder/too short, or fail-closed unverifiable strength), connect()
returned a bare False with no fatal-error info. gateway.run's reconnect
watcher treats that as transient and re-queues with backoff forever —
each retry re-instantiating the adapter and its ResponseStore sqlite
connection. Observed in production (#37011): ~501 leaked connections
(1002 fds) over ~2.5 days until EMFILE made the whole gateway
unresponsive.
Set a non-retryable fatal error (api_server_key_invalid) in connect()
when the guard rejects, covering all three rejection branches, so the
platform drops from the reconnect queue; recover with
`/platform resume api_server` after fixing the key. Same treatment as
the port-conflict guard (api_server_port_in_use, #65665 / bda8bd76a8).
Tests mirror the port-conflict precedent: each rejection path asserts
connect() is False, has_fatal_error True, fatal_error_retryable False,
and fatal_error_code api_server_key_invalid, plus a strong-key control.
Re-implementation of #38803 by @cifangyiquan against current main —
their patch targeted the old inline guard in connect() which was since
extracted to _api_key_passes_startup_guard() (and gained the
fail-closed branch in 683059feb5), so the original diff no longer
applies. Their production diagnosis and non-retryable direction
preserved.
Refs: #38803, #37011
Salvaged from PR #36180 (commits 68dfeb4b16 and 86f437509b by arimu1),
re-applied onto current main with the incidental black-reformat churn
stripped out (~1,700 lines -> the semantic change + tests).
Previously gateway/config.py enrolled the api_server platform on
`api_server_enabled or api_server_key`, so API_SERVER_ENABLED=true with
no key (or a weak/placeholder key) still loaded the platform: the
adapter is instantiated (ResponseStore/SQLite opened in __init__), the
reconnect watcher spins, and the startup guard refuses at connect() —
logging errors forever. Now the platform is enrolled only when
API_SERVER_KEY passes the same strength bar as the adapter's startup
guard (has_usable_secret, min_length=16), via a shared
_has_usable_api_server_key() helper.
The no-op `lambda cfg: True` connected-checker for API_SERVER is also
replaced with the same key check, so get_connected_platforms() only
reports the platform "up" when it could actually start.
Known limitation (intentionally out of scope): a YAML config with
`platforms.api_server.enabled: true` and no key still loads the
platform; this gate covers the env-override path only.
Dropped from the original PR: EMAIL/SMS checker additions (scope creep
beyond the PR title; absent on current main) and the wholesale black
reformat of gateway/config.py and tests.
Fixes#36111
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Widen #48476's fence guarantees to the two splitters that still emitted
fence-broken chunks:
* GatewayStreamConsumer._split_text_chunks (fallback final send): close
the orphaned ``` at each chunk boundary and reopen it — with the
original language tag — on the next chunk, mirroring
BasePlatformAdapter.truncate_message's contract. Headroom is reserved
so balanced chunks stay within the platform limit.
* Slack block_kit._split_text (3000-char section chunking): same
close/reopen balancing for mrkdwn section text carrying fences.
With these, every chunk boundary — non-streaming send
(truncate_message), streaming overflow (_truncate_for_stream via
adapter.truncate_message per #45938), fallback final
(_split_text_chunks), final-send balance (ensure_closed_code_fences),
and Block Kit section splits — delivers fence-balanced chunks.
Regression tests probe each path with fenced fixtures, assert per-chunk
balance, limit compliance, language-tag reopening, and prose passthrough.
ensure_closed_code_fences previously only handled triple-backtick
(```) code-block fences. Single backtick (`) inline-code spans have
the same problem: an orphaned opening backtick causes the remainder
of the message to render as inline code on Discord and other platforms.
After balancing triple-backtick fences, strip complete ```...```
regions and count remaining standalone backtick markers. If odd,
append a closing backtick. Same trade-off as the triple-backtick fix:
a stray closing backtick may create a brief empty inline-code span,
which is far less harmful than the rest of the message being inline code.
