Commit graph

129 commits

Author SHA1 Message Date
Brooklyn Nicholson
d2cb318509 fix(api_server): reconcile SessionDB offload with per-profile DB routing
Rebased onto current main, where _ensure_session_db grew a per-profile
cache (get_hermes_home()-keyed) for /p/<profile>/ multiplex — after this
PR's base. The PR's async rewrite assumed the old single-self._session_db
model, so on current main `session_db=self._session_db` in _create_agent
would pass None in production (the real DB lives in the per-home cache).

Split the concern: keep a SYNC _ensure_session_db (per-profile, used by
_create_agent + the many sync-patching create_agent tests) and add an
async _ensure_session_db_async that captures the profile home on the loop
thread then offloads only the SQLite open via to_thread (single-flight).
Both share _open_and_cache_session_db. Request handlers use the async
variant; _create_agent reverts to the sync call. Updated the first-request
test's FakeDB to accept db_path to match main's SessionDB(db_path=...).

Co-authored-by: necoweb3 <sswdarius@gmail.com>
2026-07-18 23:14:57 -04:00
dsad
5529175084 fix: TOCTOU race in session create + offload SessionDB init
- Make create sequence (check + insert + title) atomic via single
  _execute_write call with BEGIN IMMEDIATE, closing the TOCTOU window
  where two concurrent same-ID creates could both return 201.

- Offload _ensure_session_db() to asyncio.to_thread with single-flight
  lock so first-request SQLite init doesn't block the event loop.

- Add concurrent same-ID create test (one 201, one 409) and
  first-request path test covering the initialization.
2026-07-18 23:07:29 -04:00
dsad
7ba944d054 fix(api_server): offload synchronous SessionDB calls off the event loop 2026-07-18 23:06:17 -04:00
Teknium
9fc0074bac fix(gateway): unify reset boundaries vs recovery — promote accidental ends, honor mode=none, adapter-aware resume guidance
Unifies the two gateway subsystems that were fighting each other: the
'never lose a session' recovery machinery (#54878 stale-route self-heal,
find_latest_gateway_session_for_peer reopening agent_close/ws_orphan_reap
rows) and the session reset/expiry machinery (expiry watcher, /new,
/resume, resume_pending freshness gate).

The unified contract:
- INTENTIONAL boundaries (expiry finalization, auto-reset, /new,
  /resume switch) are recorded durably via promote_to_session_reset(),
  which upgrades accidental recoverable end_reasons (agent_close,
  ws_orphan_reap) to the explicit boundary while preserving other
  explicit reasons (compression, etc.). Recovery then correctly refuses
  to resurrect them.
- ACCIDENTAL ends (crash, cleanup bug, mistaken reaper) stay
  recoverable — genuine crash recovery is untouched.

On top of the cherry-picked contributor commits:
- promote_to_session_reset widened to ws_orphan_reap + parameterized
  reason so auto-reset paths stay auditable (idle/daily/suspended/
  resume_pending_expired) (#61220, #61993, #63539)
- get_or_create_session auto-reset, reset_session (/new), and
  switch_session (/resume) all write through the promote path — the
  first-reason-wins end_session no-op could previously leave a reset
  session resurrectable behind a stale agent_close row (#61993)
- resume_pending freshness gate now honors session_reset.mode=none:
  explicit opt-out of automatic resets also opts out of the zombie
  gate (#61052)
- resume recovery note extracted to build_resume_recovery_note() and
  made adapter-aware via a new interactive_resume capability flag:
  webhook/api_server auto-resume turns now CONTINUE the interrupted
  task instead of emitting an unanswerable 'session restored'
  acknowledgement that abandoned the work (#57056)
- tests updated to call the real note builder instead of mirroring it

E2E-validated against a real SessionDB + SessionStore in a temp
HERMES_HOME: expiry->agent_close->no-resurrection, /new promote,
crash recovery preserved, mode=none opt-out, routing-table flag sync.
2026-07-17 04:51:39 -07:00
Teknium
14f023cd00 fix(api_server): run platform event verifiers off-loop and fail closed
The platform callback verifier can do blocking network I/O (e.g. the
google-chat adapter fetches Google signing certs on a cache miss), which
would stall the event loop if called inline. Run sync verifiers via
asyncio.to_thread (await coroutine verifiers directly), and treat a
crashing verifier as a 401 rather than a 500 through the dispatch path —
a broken verifier must never admit an event.
2026-07-16 07:31:05 -07:00
aeyeopsdev
1305a690e0 feat(gateway): route platform HTTP event callbacks 2026-07-16 07:31:05 -07:00
Drexuxux
efb6c21498 fix(api-server): reject a non-ASCII bearer token with 401 instead of crashing
_check_auth gates every OpenAI-compatible API server endpoint. It compared
the client's raw bearer token against the configured key with
hmac.compare_digest on two str values. compare_digest raises TypeError on
a str containing non-ASCII characters, and the token comes straight from
the Authorization header — so a request with a single non-ASCII byte in
the key (a stray unicode char, a smart quote, a pasted BOM) crashed the
handler with an unhandled TypeError. Every endpoint calls _check_auth
without a try/except, so the framework turned that into a 500 Internal
Server Error instead of the intended 401 Invalid API key.

Compare as bytes, matching web_server.py's dashboard-token check
(hmac.compare_digest(auth.encode(), expected.encode())). Encoding both
sides keeps the timing-safe comparison and its semantics identical for
valid keys while making a non-ASCII token fail closed with a clean 401.

Adds regression tests: a non-ASCII bearer token returns 401 (no raise),
and a non-ASCII configured key still authenticates against its exact
value.
2026-07-16 07:22:24 -07:00
giggling-ginger
6ff65c4d20 fix(gateway): scope default-listener api_server requests under multiplex
Rebuilt from PR #61283 onto the /p/<profile>/ routing world (7aa21e336):
_profile_scope(None) now enters the DEFAULT profile's runtime scope when
multiplexing is active instead of returning nullcontext(). api_server is
a port-binding platform living on the default profile, so plain requests
(no /p/ prefix) are the primary path — with fail-closed get_secret they
crashed with UnscopedSecretError on the first credential read (#61276).

All three wrapped call sites (chat-completions executor, /v1/runs agent
construction and _run_sync) inherit the fix through the one seam.
Single-profile gateways keep the no-op. Regression tests ported from
the original PR to the _profile_scope seam.

Fixes #61276
2026-07-16 07:17:55 -07:00
Teknium
bda8bd76a8
fix(api_server): mark port conflict as non-retryable to stop infinite reconnect loop (#65665)
A bare False from connect() on EADDRINUSE made the gateway reconnect
watcher treat a port conflict as transient and retry forever at the
backoff cap — 1568+ retries over 5 days in a multi-profile production
setup, filling errors.log and leaking 2 ResponseStore fds per retry.
Set a non-retryable fatal error (api_server_port_in_use) in the bind
OSError branch so the platform drops from the reconnect queue;
recover via /platform resume api_server after changing the port.

Re-implementation of #52132 by @msalles1 against the direct-bind path
from #65621 (the pre-probe block their patch targeted no longer
exists). Their production diagnosis and test scenario preserved.
2026-07-16 06:29:09 -07:00
Teknium
164bca658e
fix(gateway): bind api_server directly instead of pre-probing 127.0.0.1 (#65621)
The single-family pre-probe (_port_is_available) raced the real bind and
reported a lingering TIME_WAIT socket as 'in use', failing gateway
restarts for up to ~60s (#10297). Port the webhook adapter's bind
mechanics (#63711/#65482): delete the probe, bind directly with clean
OSError handling and runner teardown, and scope reuse_address=False to
macOS only so Linux restarts rebind past TIME_WAIT instantly.

Credit to @lrawnsley (#10297) for identifying the TIME_WAIT restart
failure.
2026-07-16 05:39:43 -07:00
HexLab98
7aa21e3362 fix(gateway): add api_server /p/<profile>/ multiplex routing
Docs and MultiplexConfigError already promise secondary profiles are
served through the shared listener's /p/<profile>/ prefix, but only the
webhook adapter registered those routes — api_server returned 404.
Mirror every HTTP route, validate the prefix, and scope agent runs /
session DB / model listing to the target profile.
2026-07-16 15:59:45 +05:30
kshitijk4poor
915f1bf1bc fix(api): reserve cron fire work during drain 2026-07-13 23:26:10 +05:30
kshitijk4poor
ffc10cc659 fix(gateway): quiesce API and cron work during drains
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
2026-07-13 23:26:10 +05:30
kshitijk4poor
104ffeae23 fix(gateway): complete API-server shutdown drain 2026-07-13 23:26:10 +05:30
Bartok9
021ee34546 fix(gateway): drain in-flight api_server runs on shutdown
Closes #63529

Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.

Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.

Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
2026-07-13 23:26:10 +05:30
Teknium
d48bf743f2 fix(approval): scope smart deny owner overrides to one operation
Co-authored-by: Sergei Ivanov <kavi@local.hermes>
2026-07-13 04:31:55 -07:00
teknium1
837077dfae fix(api): stop producers after run transport expires 2026-07-12 04:15:47 -07:00
teknium1
8f18fa104f fix(api): separate run control from stream lifetime 2026-07-12 04:15:47 -07:00
teknium1
1da89a5f3d fix(api): keep live runs tracked past stream ttl 2026-07-12 04:15:47 -07:00
teknium1
6142203bd7 fix(gateway): ground readiness in live runtime state 2026-07-11 08:42:21 -07:00
teknium1
f9728af5e2 feat(gateway): add authenticated runtime readiness checks 2026-07-11 08:42:21 -07:00
Gutslabs
2b4ec0082a
fix(api_server): return 413 for oversized chunked bodies
api_server already caps every read via client_max_size (chunked
included), but when the limit tripped mid-read the handler's broad JSON
except turned it into 400 'Invalid JSON'. Catch
HTTPRequestEntityTooLarge in body_limit_middleware and return the
OpenAI-style 413.

Status-code polish extracted from PR #3949 by @Gutslabs — the PR's core
client_max_size change already exists on main.
2026-07-04 15:35:05 -07:00
srojk34
16332af60b security(gateway): anchor api_server MEDIA tag resolution to safe paths
_resolve_media_to_data_urls's ad-hoc _MEDIA_TAG_RE matched any bare
token after MEDIA: (no absolute-path anchor) and read the resolved
path directly with no denylist. A relative/traversal path like
MEDIA:../../../../etc/passwd.png slipped through, and any image-
suffixed file the process could read (including under ~/.ssh, ~/.aws,
etc.) was base64-inlined into the API response if its path merely
appeared in the model's own final reply text.

Every other platform adapter's MEDIA: handling already goes through
two shared primitives in gateway/platforms/base.py:
  - MEDIA_TAG_CLEANUP_RE, which anchors the path to ~/, /, or a
    Windows drive letter plus a known deliverable extension.
  - validate_media_delivery_path, which resolves symlinks and rejects
    paths under the credential/system-path denylist.

Reuse both here instead of the local unanchored pattern and naive
Path().expanduser() resolution.
2026-07-03 03:27:47 -07:00
Mibayy
4a09b692ec feat(api-server): per-client model routing via model_routes (#3176 salvage)
Adds a no-code routing layer to the OpenAI-compatible API server so one
Hermes deployment can map different API clients to different
model/provider backends. Clients pick a backend by sending a configured
alias as the OpenAI 'model' field; unmatched values fall back to the
global model. Configured aliases are listed by GET /v1/models.

Precedence (highest first): session /model override > model_routes
route > global config. Route provider credentials resolve through
_resolve_runtime_agent_kwargs_for_provider (same seam as
channel_overrides); per-route api_key/base_url are upstream provider
credential overrides — never caller auth, never logged.

Salvaged and rebased from PR #3176 by @Mibayy onto current main.
2026-07-02 05:23:28 -07:00
Tarun Ravikumar
2068754d6f feat(api-server): inline MEDIA: image tags as base64 data URLs for remote frontends
Salvage of the surviving piece of #2696 by @tarunravi. The PR's other two
changes (tool progress streaming, SSE None-sentinel fix) were independently
superseded on main by the structured hermes.tool.progress SSE events and the
rewritten queue-drain loop.

Remote OpenAI-compatible frontends can't read server-local file paths, so
MEDIA:<path> tags (browser screenshots, generated images) were dead text.
_resolve_media_to_data_urls() now inlines small (<=5MB) local images as
markdown data URLs across all four response surfaces: chat completions
(non-streaming), session chat, session chat stream final event, and the
Responses API. Non-image, missing, or oversized paths pass through
untouched.
2026-07-02 03:23:44 -07:00
SahilRakhaiya05
2d8d08cae6 fix(api-server): require auth for /health/detailed and fail closed on weak keys
/health/detailed leaked runtime state (gateway state, connected
platforms, active-agent counts, PID, exit reason) with no auth. Gate it
behind the same Bearer auth as other API routes; plain /health stays
open for liveness probes.

Also refuse to start on a placeholder/too-short (<16 char) API_SERVER_KEY
regardless of bind address — a guessable key on a terminal-capable
endpoint is RCE-adjacent even on loopback, since any local process can
reach it. The required-key check was already unconditional; this extends
the strength floor to loopback binds too. Startup guards are hoisted
above app/background-task creation so a rejected start leaves no partial
state.

Salvaged from #44073 (external-surface hardening), split into a focused
PR per maintainer request.

Co-authored-by: Hermes Agent <agent@nousresearch.com>
2026-07-01 04:14:33 -07:00
ryo-solo
d578b6165d fix(api_server): pop fallback model kwarg to prevent AIAgent collision
When the primary provider's auth fails (expired token / 429 quota cap),
_resolve_runtime_agent_kwargs() falls through to the fallback provider
chain, whose runtime dict carries its own 'model' key. api_server's
_create_agent then did AIAgent(model=model, **runtime_kwargs), colliding
on 'model' and 500ing every /v1/chat/completions request while a fallback
was active. Pop the runtime model and let it override the config model,
mirroring the native gateway path (_resolve_session_agent_runtime).

Salvaged from #35716 by @ryo-solo (earliest submitter); the PR's second
half (Mistral reasoning_content strip) is already handled on main and
dropped.

Co-authored-by: Hermes Agent <noreply@nousresearch.com>
2026-07-01 01:26:27 -07:00
hinotoi-agent
66325a7700 fix(api-server): scope run approvals by run id 2026-07-01 00:42:42 -07:00
teknium1
ea1372d2af fix(security): wire session-id sanitizer into artifact paths + API boundary
Defense-in-depth on top of _safe_session_filename_component (#5958):

Sink (makes the bad write impossible regardless of entry point):
- run_agent._save_session_log: sanitize session_id before building the
  session_{sid}.json snapshot path.
- agent_runtime_helpers.dump_api_request_debug: sanitize before building
  the request_dump_{sid}_{ts}.json path.

Boundary (clean 400 instead of a silently-hashed filename):
- api_server rejects path-traversal-shaped X-Hermes-Session-Id on the
  session-continuation path and the explicit /api/sessions create path,
  reusing gateway.session._is_path_unsafe (mirrors the native gateway's
  entry-boundary guard). Also enforces the session-header length cap on
  the continuation path.

Tests: traversal session_id stays contained at the write site; sanitizer
always yields a traversal-free segment; the API header rejects
../, absolute, and Windows-traversal IDs with 400.
2026-06-29 04:25:45 -07:00
teknium1
58c36b1798 fix(api-server): widen error redaction to cron-endpoint + SSE sites
Follow-up to the salvaged #37733 fix. The contributor centralized
redaction at _openai_error and the chat/responses failure paths, which
covers the OpenAI-compatible envelopes transitively. Two sibling classes
crossed the same authenticated HTTP boundary unredacted:

- 8x cron-management endpoints returning {"error": str(e)} on 500
- the session-chat SSE error event ({"message": str(exc)})

Route both through the same _redact_api_error_text(force=True) helper.
Add AUTHOR_MAP entry for coygeek and a TestRedactApiErrorText guard
covering mask/force/limit/passthrough behavior.
2026-06-28 02:05:38 -07:00
Coy Geek
5e774de76e fix(api-server): redact provider errors at HTTP boundary
Force API-server error text through the existing secret redactor before returning OpenAI-compatible errors, response fallback text, response snapshots, and run failure events. This prevents credential-shaped provider failure text from crossing the API-server boundary while preserving debuggable sanitized messages.
2026-06-28 02:05:38 -07:00
flobo3
b8b695e2cd fix(api): surface agent crash in SSE chat completions stream 2026-06-27 03:52:44 -07:00
teknium1
43b8ba4181 fix(telegram): preserve Bot API update queue on watcher reconnect
After a prolonged outage the in-process network-error ladder escalates to
fatal and GatewayRunner._platform_reconnect_watcher rebuilds a fresh adapter
that reconnects through the bootstrap path. That path called
start_polling(drop_pending_updates=True), discarding every update Telegram
queued during the outage — all messages sent while the bot was down were
silently lost. The in-process ladder and 409-conflict handler already passed
drop_pending_updates=False; only bootstrap did not distinguish a cold first
boot from a reconnect.

Thread an is_reconnect signal from the watcher through
_connect_adapter_with_timeout into adapter.connect(). The base
BasePlatformAdapter.connect() gains a keyword-only is_reconnect=False so every
adapter inherits a tolerant signature (no per-platform breakage when the
runner forwards the kwarg). Telegram translates is_reconnect into
drop_pending_updates=not is_reconnect on both the polling and webhook bootstrap
calls. Cold boot still drops the stale queue; a watcher reconnect preserves it.

Fixes #46621.

Co-authored-by: annguyenNous <annguyen@nousresearch.com>
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: Kewe63 <Kewe63@users.noreply.github.com>
2026-06-25 21:29:57 -07:00
uperLu
0d4cecb352 fix(cron): avoid provider package shadowing core cron 2026-06-23 23:39:22 -07:00
kshitij
1f28b1a9b9
fix(gateway): redact credentials from approval prompts before sending to clients (#48456) (#50767)
Tirith redacts its own findings, but the approval-request callbacks built the
operator prompt from the RAW command string, so a credential-shaped value
Tirith flagged was sent verbatim to clients, undoing the redaction one layer up.

Two egress transports carried the leak; both are fixed via a shared
module-level seam _redact_approval_command() (redact_sensitive_text force=True):
  1. chat platforms — _approval_notify_sync (gateway/run.py): redact before
     both the button path (send_exec_approval) and the plain-text /approve
     fallback.
  2. SSE/API stream — _approval_notify (gateway/platforms/api_server.py):
     redact event['command'] before it is enqueued to API/desktop clients.
     (whole-bug-class: sibling call path on a separate transport.)

force=True so the prompt — a hard secret-egress boundary — honors redaction
even when security.redact_secrets is off. Clean commands pass through unchanged.

Tests bind the seam (synthetic credential-format fixtures, force-when-disabled) AND assert
BOTH callbacks ASSIGN the redacted result before the send/enqueue sink, via an
AST contract that rejects a discarded-result call. All mutation-checked.
2026-06-22 11:39:45 +00:00
teknium1
7726ce3040 fix(security): close hermes-0day MCP-persistence attack surface
Remove the dashboard --insecure auth-bypass, add an MCP persistence guard +
IOC blocklist, and raise the API-server key entropy floor.

Driven by the June 2026 hermes-0day campaign (r/hermesagent, live 854.media
instance): scanners find exposed Hermes dashboards/API servers, drive the
root agent to plant a 'command: bash' MCP entry that appends an attacker SSH
key to authorized_keys, which cron + startup then re-execute every tick.

- dashboard: --insecure no longer disables the auth gate. should_require_auth
  returns True for every non-loopback bind; a public bind ALWAYS requires an
  auth provider (bundled password provider or OAuth). --insecure kept as a
  warned no-op for backward compat. Fail-closed error now points at the
  password provider, not at --insecure.
- mcp_security: validate_mcp_server_entry now also rejects shell payloads that
  write to OS persistence surfaces (authorized_keys/.ssh/pam.d/sudoers/cron/
  rc files) and hard-rejects a hermes-0day IOC blocklist (attacker SSH key +
  source IPs) anywhere in command/args/env. Runs at save AND spawn time.
- api_server: raise network-bind API_SERVER_KEY entropy floor 8->16 chars;
  warn when a network-accessible API server runs an unsandboxed local backend.
2026-06-21 19:05:27 -07:00
Teknium
7a131f7f40
fix(api-server): stop silently promising async delivery on stateless HTTP path (#50319)
* fix(api-server): stop silently promising async delivery on stateless HTTP path

terminal(notify_on_complete=True / watch_patterns) and delegate_task(background=True)
silently no-op'd on the API server / WebUI path (#10760): the watcher / detached
child registered, but every API-server route (OpenAI-spec /v1/chat/completions
and /v1/responses, plus the proprietary /v1/runs SSE stream) tears down its
channel when the turn ends, and APIServerAdapter.send() is a no-op stub. A
completion that fires after the response closed had nowhere to go — from the
agent side, indistinguishable from a hang.

There is no spec-compliant surface to wake the agent later on a stateless HTTP
client, so make the no-op honest instead of silent:

- Add a per-adapter capability flag supports_async_delivery (default True;
  APIServerAdapter = False), propagated into a HERMES_SESSION_ASYNC_DELIVERY
  contextvar via async_delivery_supported(). Toggle on the adapter, not a
  hardcoded platform string — a future stateless adapter is correct-by-default.
- terminal: when delivery is unsupported, skip watcher registration, force
  notify_on_complete off, and return a notify_unsupported note telling the
  agent to process(action='poll').
- delegate_task: when delivery is unsupported, fall back to SYNCHRONOUS
  execution (work runs and returns in the same response) with a note, instead
  of handing out a handle that never resolves.

CLI (in-process completion_queue) and the real gateway platforms are unchanged.

Fixes #10760

* refactor(api-server): route session binding through a single no-delivery chokepoint

Add APIServerAdapter._bind_api_server_session() and route both agent-entry
paths (_run_agent for /v1/chat/completions + /v1/responses, and the /v1/runs
_run_sync path) through it. The helper hardwires platform="api_server" and
async_delivery=False with no async_delivery parameter to pass, so a future
route added to the API server physically cannot reintroduce the silent
no-op (#10760) by forgetting to mark the channel as non-delivering.

The binding stays request-scoped (cleared per turn), so a session resumed
later on a delivering interface (CLI / gateway platform) re-binds fresh and
is NOT blocked — the no-delivery decision tracks the interface handling the
current turn, never the session.
2026-06-21 12:15:14 -07:00
Teknium
e499d69e3e
feat(api-server): configurable concurrent-run cap to prevent DoS (#50007)
The OpenAI-compatible API server only enforced a hardcoded cap of 10
concurrent runs on /v1/runs, leaving /v1/chat/completions and
/v1/responses unbounded — a request flood could exhaust CPU, memory,
and upstream LLM quota (#7483).

- Add gateway.api_server.max_concurrent_runs (config.yaml, default 10,
  0 disables). No env var.
- Shared concurrency gate across all three agent-serving endpoints,
  counting both the chat/responses in-flight counter and the /v1/runs
  stream set. Returns OpenAI-style 429 + Retry-After when at the cap.
- Remove the dead hardcoded _MAX_CONCURRENT_RUNS class attribute.

Closes #7483.
2026-06-21 07:26:03 -07:00
kshitijk4poor
b577f25100 refactor(gateway): dedupe drain-timeout resolution + share active_agents parse
Follow-up cleanups on top of the busy/idle readout (PR #50103):

- web_server.py /api/status reused the single drain-timeout resolver
  hermes_cli.gateway._get_restart_drain_timeout() (HERMES_RESTART_DRAIN_TIMEOUT
  env -> agent.restart_drain_timeout config -> default) instead of inlining a
  third hand-rolled copy of that precedence chain. Also fixes a subtle
  divergence: the inline copy used os.environ.get() so a set-but-empty env var
  was treated as a value rather than falling through to config; the shared
  resolver .strip()s and falls through correctly.
- Added gateway.status.parse_active_agents() and routed BOTH HTTP surfaces
  (/api/status and /health/detailed) through it, so the exposed active_agents
  field is consistently clamped non-negative. Previously /api/status clamped
  while /health/detailed exposed the raw file value, diverging on a corrupt
  count.
- Added TestParseActiveAgents covering the shared coercion contract.
2026-06-21 17:22:52 +05:30
Ben
0ee75469d7 feat(dashboard): surface gateway busy/drainable on /api/status
Give an external consumer (NAS) a trustworthy, always-reachable busy/idle
readout it can poll before a disruptive lifecycle action (restart,
migrate, stop, auto-update). The dashboard /api/status is the only HTTP
surface guaranteed up on a hosted agent regardless of which gateway
platforms are enabled, and it already reads gateway_state.json.

Add to /api/status (additive, non-breaking):
  - active_agents       — in-flight gateway-turn count (now refreshed
                          per-turn by the companion gateway-side commit)
  - gateway_busy        — running AND active_agents > 0
  - gateway_drainable   — running and live (a valid begin-drain target)
  - restart_drain_timeout — resolved seconds, so the consumer can size its
                          poll deadline without out-of-band knowledge
                          (env HERMES_RESTART_DRAIN_TIMEOUT → config
                          agent.restart_drain_timeout → default)

The busy/drainable contract is defined once in gateway.status
(derive_gateway_busy / derive_gateway_drainable) and consumed by both
/api/status and /health/detailed so the two surfaces can never disagree.
Liveness keys off gateway_running (a live PID/health probe), NEVER
gateway_updated_at — a healthy idle gateway never advances that timestamp.
All derived fields degrade to safe falsy values when the gateway is down
or the status file is absent/corrupt (never a spurious "busy" that would
wedge the consumer). active_sessions (the 5-min DB recency heuristic the
SPA reads) is left exactly as-is — new signal, new fields.

Tests (behaviour contracts, not snapshots): the pure derivation contract
across every running/state/count/liveness combination; /api/status
integration for busy, idle-drainable, draining, down, stale-busy-file,
corrupt-count, and timeout surfacing; and /health/detailed parity.
2026-06-21 17:22:52 +05:30
teknium1
a58287afcb
Merge remote-tracking branch 'origin/main' into pr48275-rebase
# Conflicts:
#	cron/scheduler.py
2026-06-19 07:40:29 -07:00
infinitycrew39
460b1e50e5 fix(gateway): refresh max_turns before resolving runtime budget 2026-06-19 06:31:13 -07:00
Ben
b75757d4aa feat(cron): wire on_jobs_changed, cron.chronos config, docs + agent↔NAS contract
Phase 4F (F.1 + F.2 + F.3, agent side). F.4 is the operator-run live smoke
(needs a NAS deployment); recorded in the PR, not code.

F.1 — on_jobs_changed wiring:
- cron/scheduler.py: _notify_provider_jobs_changed() — resolve the active
  provider, call on_jobs_changed(), swallow errors. Lives in scheduler.py (not
  jobs.py) so the store stays free of provider imports (no import cycle).
- Wired at the consumer surfaces AFTER a successful mutation: the cronjob model
  tool (tools/cronjob_tools.py, create/update/remove/pause/resume) — which the
  `hermes cron` CLI also routes through — and the REST handlers
  (gateway/platforms/api_server.py, same five). Built-in's no-op default = zero
  behavior change on the default path. Sleeping-agent direct jobs.json writes
  (no tool/CLI/REST) are covered by reconcile-on-wake in start().

F.2 — config: cron.chronos.{portal_url,callback_url,expected_audience,
nas_jwks_url}. All non-secret; the agent holds no scheduler creds and the
outbound provision call reuses the existing Nous token (no token key). Additive
deep-merge key, no version literal.

F.3 — docs:
- docs/chronos-managed-cron-contract.md: authoritative agent↔NAS wire contract
  (the three agent-cron endpoints + inbound /api/cron/fire + the 3-hop trust
  model + at-most-once/re-arm semantics). This is what the NAS-side agent builds
  against.
- cron-internals.md: "Managed cron (Chronos) for scale-to-zero" section.
- cli-commands.md: cron.provider accepts chronos + the cron.chronos.* keys.
- User docs name no scheduler vendor (QStash is a NAS-internal detail).

INVARIANT re-verified: zero qstash/upstash hits across plugins/cron, gateway,
hermes_cli, tools, website/docs (the one remaining repo hit is an unrelated
Context7 MCP comment in tools/mcp_tool.py).

Tests: test_jobs_changed_notify (5) — notify calls provider hook, swallows
errors, built-in harmless, tool create/remove notify. Full cron + chronos +
webhook + config + api_server_jobs suites green (504 in the cron+chronos+webhook
run).
2026-06-18 15:11:32 +10:00
Ben
3fc7b624d8 feat(cron,gateway): NAS-JWT fire verifier + /api/cron/fire webhook (Chronos)
Phase 4E (E.1 + E.2). The inbound side of Chronos: NAS POSTs the agent when a
one-shot fires; the agent verifies a NAS-minted JWT and runs the job.

E.1 — plugins/cron/chronos/verify.py:
- verify_nas_fire_token(token, expected_audience, jwks_or_key, issuer): verifies
  signature against the NAS JWKS (RS/ES family; symmetric rejected), aud == this
  agent, exp/nbf, iss, and purpose == "cron_fire" (so a general agent JWT can't
  be replayed against the fire endpoint). Returns claims or None; never raises.
  Crypto delegated to PyJWT[crypto] (already a declared dep) — no hand-rolled
  JWT, no new dependency. No key configured → refuse (never unsigned-decode a
  security boundary).
- get_fire_verifier(): pluggable indirection so the DQ-4 escape hatch
  (direct per-job cron-key) can swap in with no handler change.

E.2 — gateway/platforms/api_server.py:
- POST /api/cron/fire (registered only when _CRON_AVAILABLE). Authenticated by
  the NAS-JWT via get_fire_verifier() — NOT API_SERVER_KEY (NAS holds no API
  key; this is the only inbound that triggers remote job execution, so it gets
  its own purpose-scoped check). Verifier args come from cron.chronos.* config.
  401 on bad/missing/forged token. 400 on missing job_id. On success: 202 +
  fire_due runs in the background (so a long agent turn never trips NAS's HTTP
  timeout); the store CAS claim inside fire_due de-dupes a scheduler retry.

Tests:
- test_chronos_verify (11): REAL RS256 signing — valid→claims, wrong-aud,
  missing/wrong purpose, expired, wrong-iss, tampered-signature (attacker key),
  no-key-refuse, empty-token, JWKS-URL key resolution, get_fire_verifier.
- test_cron_fire_webhook (5): valid→202+fire, invalid→401+no-fire, missing
  token→401, missing job_id→400, and fire path does NOT require API_SERVER_KEY.
api_server regression suites (214) green.

E.3 (NAS endpoints) is a separate cross-repo PR; the wire contract lands next
(docs/chronos-managed-cron-contract.md).
2026-06-18 14:46:33 +10:00
Teknium
5105c3651a
perf(api-server): normalize chat content linearly (#46079) 2026-06-14 03:25:49 -07:00
helix4u
b23184cad4 fix(api-server): bind request session context for tools 2026-06-08 20:52:08 -07:00
konsisumer
3714caa1b9 fix(session): follow compression continuations for transcript reads 2026-06-07 23:57:20 -07:00
Teknium
30c7913617
fix(api_server): report hermes version on /health and /health/detailed (#40620)
Salvaged from #40479; re-verified on main, tightened, tested.

Co-authored-by: tfournet <tfournet@users.noreply.github.com>
2026-06-07 18:38:54 -07:00
Teknium
0c48b7165d hardening(api-server): scan cron prompts on REST create/update for parity with the agent tool
The agent-facing cronjob tool scans the user prompt with _scan_cron_prompt()
before creating/updating a job (tools/cronjob_tools.py); the REST cron
endpoints (POST /api/jobs, PATCH /api/jobs/{id}) validated length but not
content. This adds the same scan to both handlers so an exfiltration/injection
prompt is rejected the same way regardless of which surface created the job.

NOT a security boundary, defense-in-depth / parity only: the REST cron
endpoints are authenticated (every handler runs _check_auth, and connect()
refuses to start without API_SERVER_KEY), and _scan_cron_prompt is a documented
in-process heuristic, not a containment boundary (SECURITY.md 3.2).

Raised externally via GHSA-fr3q-rjg3-x6mf (DNS-rebinding pre-auth RCE). The
report's load-bearing 'no auth by default' premise was already closed three
weeks after it was filed by the API_SERVER_KEY-required guard (commit
1a9ef8314); this lands the create/update prompt-validation parity the report
also pointed at. Scanner imported defensively so a missing scanner cannot
disable the cron REST API.
2026-06-07 10:04:57 -07:00
annguyenNous
f7dabd3019 fix(api-server): guard json.loads against corrupted SQLite data in response cache
The ResponseStore.get() method calls json.loads(row[0]) without any
error handling. If the SQLite responses table contains corrupted JSON
data (e.g. from a crash mid-write or disk corruption), this raises
an unhandled JSONDecodeError that propagates to the caller.

Fix: wrap in try/except (json.JSONDecodeError, TypeError). On parse
failure, log a warning, evict the corrupted entry from the cache, and
return None (consistent with the function's Optional return type).
2026-06-04 06:15:29 -07:00