The gateway /compress path can force a user-leading layout that leaves
the compaction summary after a retained system head, so scanning only
the leading block misses it. Preserve marker-carrying messages wherever
they sit, filling the remaining budget with the most recent other
messages, and cover the non-leading position with a test.
Rework on top of the salvaged #58133 commits:
- Remove the compression.persist_in_response_store config key — this is
a bug fix (stored transcripts must reflect what the agent will actually
replay), not behavior that should be opt-out-able.
- Drop the per-request load_config() imports the handler-level persist
blocks added.
- Dedupe the two handler-level persist blocks: the compressed-transcript
substitution already lives in _build_response_conversation_history
(via result["_compressed"]), so the handlers only need to propagate
the effective (possibly rotation-changed) session_id. The streaming
path does this via a new session_id_snapshot arg on
_persist_response_snapshot; the non-streaming path picks up
result["session_id"] directly.
- Rotation propagation no longer gates on history-from-store: the first
request in a chain can also rotate, and its stored session_id must be
the child session or the next previous_response_id request resumes the
pre-rotation session and re-compresses every turn.
The persist logic only checked _result_sid != session_id (rotation),
missing in-place mode where session_id is unchanged but _compressed
flag is set. response_store history doubled every turn (11->26->55->110->225)
causing repeated re-compression.
Fix: detect compression via _did_compress or _rotated, and only update
_effective_session_id on actual rotation (not in-place).
Note: preflight loop break (turn_context.py) from original commit
eee64097a is excluded — it's an optimization, not a bug fix.
Cherry-picked from alidev eee64097a (api_server.py only)
- Detect when history is loaded from response_store (via previous_response_id)
- Add history_from_store parameter to distinguish history source
- When compression occurs, persist compressed messages instead of original
- Add persist_in_response_store config option (default True)
- Update session_id and response headers to reflect session rotation
Cherry-picked from alidev 2eb816f6b
Compression produces a compact transcript in result['messages'],
but _build_response_conversation_history detected a prefix mismatch
and concatenated the original conversation_history on front.
Detect compression via _last_compaction_in_place / session_id
rotation and signal through result['_compressed'] so the builder
uses the compressed transcript directly.
The /api/status gateway_updated_at field and the gateway /health/detailed
updated_at field passed through whatever gateway_state.json contained,
untyped. All current writers emit RFC3339 via _utc_now_iso(), but legacy
gateways wrote unix epoch floats, and a corrupt or hand-edited state file
can inject numbers or arbitrary garbage — while the frontend types
(web/src/lib/api.ts) declare string | null.
Add normalize_updated_at() in gateway/status.py as the single funnel:
- str: accepted iff datetime.fromisoformat parses (trailing Z tolerated);
naive timestamps coerced to UTC; canonical isoformat returned
- int/float: treated as unix epoch seconds -> UTC ISO string, with a
plausibility guard (reject < 2000-01-01, > now+1day, non-finite)
- bool: rejected explicitly (int subclass, but never a timestamp)
- anything else: None
Apply it at both emit sites: the dashboard /api/status handler (covers
both the local read_runtime_status() branch and the remote
/health/detailed cross-container fallback branch) and the gateway API
server's /health/detailed response.
Contract tests: parametrized /api/status normalization (epoch float/int,
garbage string, None, bool, dict, absent key), remote-health numeric and
garbage bodies, dashboard shape test round-trip assertion, direct
normalize_updated_at units (range guards, Z suffix, naive coercion,
non-finite floats), and a write_runtime_status -> read_runtime_status
round-trip proving the writer side stays tz-aware parseable.
_readiness_work_counts()'s active_api_runs set is {"queued", "running",
"waiting_for_approval"} — it excludes "stopping", the status
_handle_stop_run() sets while a run is being interrupted. Since the stop
is fully cooperative (the run stays "stopping" — doing real
executor-thread work — until the agent actually notices the interrupt and
the task settles to "cancelled", an unbounded window, not a fixed
timeout), /health/detailed's background_queues.active_api_runs
undercounts real active work for that whole duration.
Fix: add "stopping" to the active-status set. background_queues.status
itself is hardcoded "ok" (gateway/readiness.py), so this doesn't change
overall readiness — it only corrects the count value external monitoring
tooling reads from this endpoint.
/health and /health/detailed resolve the version via importlib.metadata
first, falling back to hermes_cli.__version__. On editable/source
checkouts — including the standard git-based install that hermes-setup
performs — hermes_agent-*.dist-info can survive a source update
unchanged, so the health endpoints keep reporting the previous release
even though the running code (CLI, dashboard, release tags) is newer.
Stale metadata does not raise, so the source fallback never fires.
Flip the preference: use hermes_cli.__version__ (the runtime source of
truth shared by the CLI and dashboard) first, and fall back to
distribution metadata only when the source import fails. The
never-raise contract of the version probe is unchanged.
Observed live: CLI, dashboard, and pyproject all reported 0.18.2 while
/health returned 0.18.0 from a stale hermes_agent-0.18.0.dist-info left
behind by a source update.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebased onto current main, where _ensure_session_db grew a per-profile
cache (get_hermes_home()-keyed) for /p/<profile>/ multiplex — after this
PR's base. The PR's async rewrite assumed the old single-self._session_db
model, so on current main `session_db=self._session_db` in _create_agent
would pass None in production (the real DB lives in the per-home cache).
Split the concern: keep a SYNC _ensure_session_db (per-profile, used by
_create_agent + the many sync-patching create_agent tests) and add an
async _ensure_session_db_async that captures the profile home on the loop
thread then offloads only the SQLite open via to_thread (single-flight).
Both share _open_and_cache_session_db. Request handlers use the async
variant; _create_agent reverts to the sync call. Updated the first-request
test's FakeDB to accept db_path to match main's SessionDB(db_path=...).
Co-authored-by: necoweb3 <sswdarius@gmail.com>
- Make create sequence (check + insert + title) atomic via single
_execute_write call with BEGIN IMMEDIATE, closing the TOCTOU window
where two concurrent same-ID creates could both return 201.
- Offload _ensure_session_db() to asyncio.to_thread with single-flight
lock so first-request SQLite init doesn't block the event loop.
- Add concurrent same-ID create test (one 201, one 409) and
first-request path test covering the initialization.
Unifies the two gateway subsystems that were fighting each other: the
'never lose a session' recovery machinery (#54878 stale-route self-heal,
find_latest_gateway_session_for_peer reopening agent_close/ws_orphan_reap
rows) and the session reset/expiry machinery (expiry watcher, /new,
/resume, resume_pending freshness gate).
The unified contract:
- INTENTIONAL boundaries (expiry finalization, auto-reset, /new,
/resume switch) are recorded durably via promote_to_session_reset(),
which upgrades accidental recoverable end_reasons (agent_close,
ws_orphan_reap) to the explicit boundary while preserving other
explicit reasons (compression, etc.). Recovery then correctly refuses
to resurrect them.
- ACCIDENTAL ends (crash, cleanup bug, mistaken reaper) stay
recoverable — genuine crash recovery is untouched.
On top of the cherry-picked contributor commits:
- promote_to_session_reset widened to ws_orphan_reap + parameterized
reason so auto-reset paths stay auditable (idle/daily/suspended/
resume_pending_expired) (#61220, #61993, #63539)
- get_or_create_session auto-reset, reset_session (/new), and
switch_session (/resume) all write through the promote path — the
first-reason-wins end_session no-op could previously leave a reset
session resurrectable behind a stale agent_close row (#61993)
- resume_pending freshness gate now honors session_reset.mode=none:
explicit opt-out of automatic resets also opts out of the zombie
gate (#61052)
- resume recovery note extracted to build_resume_recovery_note() and
made adapter-aware via a new interactive_resume capability flag:
webhook/api_server auto-resume turns now CONTINUE the interrupted
task instead of emitting an unanswerable 'session restored'
acknowledgement that abandoned the work (#57056)
- tests updated to call the real note builder instead of mirroring it
E2E-validated against a real SessionDB + SessionStore in a temp
HERMES_HOME: expiry->agent_close->no-resurrection, /new promote,
crash recovery preserved, mode=none opt-out, routing-table flag sync.
The platform callback verifier can do blocking network I/O (e.g. the
google-chat adapter fetches Google signing certs on a cache miss), which
would stall the event loop if called inline. Run sync verifiers via
asyncio.to_thread (await coroutine verifiers directly), and treat a
crashing verifier as a 401 rather than a 500 through the dispatch path —
a broken verifier must never admit an event.
_check_auth gates every OpenAI-compatible API server endpoint. It compared
the client's raw bearer token against the configured key with
hmac.compare_digest on two str values. compare_digest raises TypeError on
a str containing non-ASCII characters, and the token comes straight from
the Authorization header — so a request with a single non-ASCII byte in
the key (a stray unicode char, a smart quote, a pasted BOM) crashed the
handler with an unhandled TypeError. Every endpoint calls _check_auth
without a try/except, so the framework turned that into a 500 Internal
Server Error instead of the intended 401 Invalid API key.
Compare as bytes, matching web_server.py's dashboard-token check
(hmac.compare_digest(auth.encode(), expected.encode())). Encoding both
sides keeps the timing-safe comparison and its semantics identical for
valid keys while making a non-ASCII token fail closed with a clean 401.
Adds regression tests: a non-ASCII bearer token returns 401 (no raise),
and a non-ASCII configured key still authenticates against its exact
value.
Rebuilt from PR #61283 onto the /p/<profile>/ routing world (7aa21e336):
_profile_scope(None) now enters the DEFAULT profile's runtime scope when
multiplexing is active instead of returning nullcontext(). api_server is
a port-binding platform living on the default profile, so plain requests
(no /p/ prefix) are the primary path — with fail-closed get_secret they
crashed with UnscopedSecretError on the first credential read (#61276).
All three wrapped call sites (chat-completions executor, /v1/runs agent
construction and _run_sync) inherit the fix through the one seam.
Single-profile gateways keep the no-op. Regression tests ported from
the original PR to the _profile_scope seam.
Fixes#61276
A bare False from connect() on EADDRINUSE made the gateway reconnect
watcher treat a port conflict as transient and retry forever at the
backoff cap — 1568+ retries over 5 days in a multi-profile production
setup, filling errors.log and leaking 2 ResponseStore fds per retry.
Set a non-retryable fatal error (api_server_port_in_use) in the bind
OSError branch so the platform drops from the reconnect queue;
recover via /platform resume api_server after changing the port.
Re-implementation of #52132 by @msalles1 against the direct-bind path
from #65621 (the pre-probe block their patch targeted no longer
exists). Their production diagnosis and test scenario preserved.
The single-family pre-probe (_port_is_available) raced the real bind and
reported a lingering TIME_WAIT socket as 'in use', failing gateway
restarts for up to ~60s (#10297). Port the webhook adapter's bind
mechanics (#63711/#65482): delete the probe, bind directly with clean
OSError handling and runner teardown, and scope reuse_address=False to
macOS only so Linux restarts rebind past TIME_WAIT instantly.
Credit to @lrawnsley (#10297) for identifying the TIME_WAIT restart
failure.
Docs and MultiplexConfigError already promise secondary profiles are
served through the shared listener's /p/<profile>/ prefix, but only the
webhook adapter registered those routes — api_server returned 404.
Mirror every HTTP route, validate the prefix, and scope agent runs /
session DB / model listing to the target profile.
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
api_server already caps every read via client_max_size (chunked
included), but when the limit tripped mid-read the handler's broad JSON
except turned it into 400 'Invalid JSON'. Catch
HTTPRequestEntityTooLarge in body_limit_middleware and return the
OpenAI-style 413.
Status-code polish extracted from PR #3949 by @Gutslabs — the PR's core
client_max_size change already exists on main.
_resolve_media_to_data_urls's ad-hoc _MEDIA_TAG_RE matched any bare
token after MEDIA: (no absolute-path anchor) and read the resolved
path directly with no denylist. A relative/traversal path like
MEDIA:../../../../etc/passwd.png slipped through, and any image-
suffixed file the process could read (including under ~/.ssh, ~/.aws,
etc.) was base64-inlined into the API response if its path merely
appeared in the model's own final reply text.
Every other platform adapter's MEDIA: handling already goes through
two shared primitives in gateway/platforms/base.py:
- MEDIA_TAG_CLEANUP_RE, which anchors the path to ~/, /, or a
Windows drive letter plus a known deliverable extension.
- validate_media_delivery_path, which resolves symlinks and rejects
paths under the credential/system-path denylist.
Reuse both here instead of the local unanchored pattern and naive
Path().expanduser() resolution.
Adds a no-code routing layer to the OpenAI-compatible API server so one
Hermes deployment can map different API clients to different
model/provider backends. Clients pick a backend by sending a configured
alias as the OpenAI 'model' field; unmatched values fall back to the
global model. Configured aliases are listed by GET /v1/models.
Precedence (highest first): session /model override > model_routes
route > global config. Route provider credentials resolve through
_resolve_runtime_agent_kwargs_for_provider (same seam as
channel_overrides); per-route api_key/base_url are upstream provider
credential overrides — never caller auth, never logged.
Salvaged and rebased from PR #3176 by @Mibayy onto current main.
Salvage of the surviving piece of #2696 by @tarunravi. The PR's other two
changes (tool progress streaming, SSE None-sentinel fix) were independently
superseded on main by the structured hermes.tool.progress SSE events and the
rewritten queue-drain loop.
Remote OpenAI-compatible frontends can't read server-local file paths, so
MEDIA:<path> tags (browser screenshots, generated images) were dead text.
_resolve_media_to_data_urls() now inlines small (<=5MB) local images as
markdown data URLs across all four response surfaces: chat completions
(non-streaming), session chat, session chat stream final event, and the
Responses API. Non-image, missing, or oversized paths pass through
untouched.
/health/detailed leaked runtime state (gateway state, connected
platforms, active-agent counts, PID, exit reason) with no auth. Gate it
behind the same Bearer auth as other API routes; plain /health stays
open for liveness probes.
Also refuse to start on a placeholder/too-short (<16 char) API_SERVER_KEY
regardless of bind address — a guessable key on a terminal-capable
endpoint is RCE-adjacent even on loopback, since any local process can
reach it. The required-key check was already unconditional; this extends
the strength floor to loopback binds too. Startup guards are hoisted
above app/background-task creation so a rejected start leaves no partial
state.
Salvaged from #44073 (external-surface hardening), split into a focused
PR per maintainer request.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
When the primary provider's auth fails (expired token / 429 quota cap),
_resolve_runtime_agent_kwargs() falls through to the fallback provider
chain, whose runtime dict carries its own 'model' key. api_server's
_create_agent then did AIAgent(model=model, **runtime_kwargs), colliding
on 'model' and 500ing every /v1/chat/completions request while a fallback
was active. Pop the runtime model and let it override the config model,
mirroring the native gateway path (_resolve_session_agent_runtime).
Salvaged from #35716 by @ryo-solo (earliest submitter); the PR's second
half (Mistral reasoning_content strip) is already handled on main and
dropped.
Co-authored-by: Hermes Agent <noreply@nousresearch.com>
Defense-in-depth on top of _safe_session_filename_component (#5958):
Sink (makes the bad write impossible regardless of entry point):
- run_agent._save_session_log: sanitize session_id before building the
session_{sid}.json snapshot path.
- agent_runtime_helpers.dump_api_request_debug: sanitize before building
the request_dump_{sid}_{ts}.json path.
Boundary (clean 400 instead of a silently-hashed filename):
- api_server rejects path-traversal-shaped X-Hermes-Session-Id on the
session-continuation path and the explicit /api/sessions create path,
reusing gateway.session._is_path_unsafe (mirrors the native gateway's
entry-boundary guard). Also enforces the session-header length cap on
the continuation path.
Tests: traversal session_id stays contained at the write site; sanitizer
always yields a traversal-free segment; the API header rejects
../, absolute, and Windows-traversal IDs with 400.
Follow-up to the salvaged #37733 fix. The contributor centralized
redaction at _openai_error and the chat/responses failure paths, which
covers the OpenAI-compatible envelopes transitively. Two sibling classes
crossed the same authenticated HTTP boundary unredacted:
- 8x cron-management endpoints returning {"error": str(e)} on 500
- the session-chat SSE error event ({"message": str(exc)})
Route both through the same _redact_api_error_text(force=True) helper.
Add AUTHOR_MAP entry for coygeek and a TestRedactApiErrorText guard
covering mask/force/limit/passthrough behavior.
Force API-server error text through the existing secret redactor before returning OpenAI-compatible errors, response fallback text, response snapshots, and run failure events. This prevents credential-shaped provider failure text from crossing the API-server boundary while preserving debuggable sanitized messages.
After a prolonged outage the in-process network-error ladder escalates to
fatal and GatewayRunner._platform_reconnect_watcher rebuilds a fresh adapter
that reconnects through the bootstrap path. That path called
start_polling(drop_pending_updates=True), discarding every update Telegram
queued during the outage — all messages sent while the bot was down were
silently lost. The in-process ladder and 409-conflict handler already passed
drop_pending_updates=False; only bootstrap did not distinguish a cold first
boot from a reconnect.
Thread an is_reconnect signal from the watcher through
_connect_adapter_with_timeout into adapter.connect(). The base
BasePlatformAdapter.connect() gains a keyword-only is_reconnect=False so every
adapter inherits a tolerant signature (no per-platform breakage when the
runner forwards the kwarg). Telegram translates is_reconnect into
drop_pending_updates=not is_reconnect on both the polling and webhook bootstrap
calls. Cold boot still drops the stale queue; a watcher reconnect preserves it.
Fixes#46621.
Co-authored-by: annguyenNous <annguyen@nousresearch.com>
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: Kewe63 <Kewe63@users.noreply.github.com>
Tirith redacts its own findings, but the approval-request callbacks built the
operator prompt from the RAW command string, so a credential-shaped value
Tirith flagged was sent verbatim to clients, undoing the redaction one layer up.
Two egress transports carried the leak; both are fixed via a shared
module-level seam _redact_approval_command() (redact_sensitive_text force=True):
1. chat platforms — _approval_notify_sync (gateway/run.py): redact before
both the button path (send_exec_approval) and the plain-text /approve
fallback.
2. SSE/API stream — _approval_notify (gateway/platforms/api_server.py):
redact event['command'] before it is enqueued to API/desktop clients.
(whole-bug-class: sibling call path on a separate transport.)
force=True so the prompt — a hard secret-egress boundary — honors redaction
even when security.redact_secrets is off. Clean commands pass through unchanged.
Tests bind the seam (synthetic credential-format fixtures, force-when-disabled) AND assert
BOTH callbacks ASSIGN the redacted result before the send/enqueue sink, via an
AST contract that rejects a discarded-result call. All mutation-checked.
Remove the dashboard --insecure auth-bypass, add an MCP persistence guard +
IOC blocklist, and raise the API-server key entropy floor.
Driven by the June 2026 hermes-0day campaign (r/hermesagent, live 854.media
instance): scanners find exposed Hermes dashboards/API servers, drive the
root agent to plant a 'command: bash' MCP entry that appends an attacker SSH
key to authorized_keys, which cron + startup then re-execute every tick.
- dashboard: --insecure no longer disables the auth gate. should_require_auth
returns True for every non-loopback bind; a public bind ALWAYS requires an
auth provider (bundled password provider or OAuth). --insecure kept as a
warned no-op for backward compat. Fail-closed error now points at the
password provider, not at --insecure.
- mcp_security: validate_mcp_server_entry now also rejects shell payloads that
write to OS persistence surfaces (authorized_keys/.ssh/pam.d/sudoers/cron/
rc files) and hard-rejects a hermes-0day IOC blocklist (attacker SSH key +
source IPs) anywhere in command/args/env. Runs at save AND spawn time.
- api_server: raise network-bind API_SERVER_KEY entropy floor 8->16 chars;
warn when a network-accessible API server runs an unsandboxed local backend.
* fix(api-server): stop silently promising async delivery on stateless HTTP path
terminal(notify_on_complete=True / watch_patterns) and delegate_task(background=True)
silently no-op'd on the API server / WebUI path (#10760): the watcher / detached
child registered, but every API-server route (OpenAI-spec /v1/chat/completions
and /v1/responses, plus the proprietary /v1/runs SSE stream) tears down its
channel when the turn ends, and APIServerAdapter.send() is a no-op stub. A
completion that fires after the response closed had nowhere to go — from the
agent side, indistinguishable from a hang.
There is no spec-compliant surface to wake the agent later on a stateless HTTP
client, so make the no-op honest instead of silent:
- Add a per-adapter capability flag supports_async_delivery (default True;
APIServerAdapter = False), propagated into a HERMES_SESSION_ASYNC_DELIVERY
contextvar via async_delivery_supported(). Toggle on the adapter, not a
hardcoded platform string — a future stateless adapter is correct-by-default.
- terminal: when delivery is unsupported, skip watcher registration, force
notify_on_complete off, and return a notify_unsupported note telling the
agent to process(action='poll').
- delegate_task: when delivery is unsupported, fall back to SYNCHRONOUS
execution (work runs and returns in the same response) with a note, instead
of handing out a handle that never resolves.
CLI (in-process completion_queue) and the real gateway platforms are unchanged.
Fixes#10760
* refactor(api-server): route session binding through a single no-delivery chokepoint
Add APIServerAdapter._bind_api_server_session() and route both agent-entry
paths (_run_agent for /v1/chat/completions + /v1/responses, and the /v1/runs
_run_sync path) through it. The helper hardwires platform="api_server" and
async_delivery=False with no async_delivery parameter to pass, so a future
route added to the API server physically cannot reintroduce the silent
no-op (#10760) by forgetting to mark the channel as non-delivering.
The binding stays request-scoped (cleared per turn), so a session resumed
later on a delivering interface (CLI / gateway platform) re-binds fresh and
is NOT blocked — the no-delivery decision tracks the interface handling the
current turn, never the session.
The OpenAI-compatible API server only enforced a hardcoded cap of 10
concurrent runs on /v1/runs, leaving /v1/chat/completions and
/v1/responses unbounded — a request flood could exhaust CPU, memory,
and upstream LLM quota (#7483).
- Add gateway.api_server.max_concurrent_runs (config.yaml, default 10,
0 disables). No env var.
- Shared concurrency gate across all three agent-serving endpoints,
counting both the chat/responses in-flight counter and the /v1/runs
stream set. Returns OpenAI-style 429 + Retry-After when at the cap.
- Remove the dead hardcoded _MAX_CONCURRENT_RUNS class attribute.
Closes#7483.
Follow-up cleanups on top of the busy/idle readout (PR #50103):
- web_server.py /api/status reused the single drain-timeout resolver
hermes_cli.gateway._get_restart_drain_timeout() (HERMES_RESTART_DRAIN_TIMEOUT
env -> agent.restart_drain_timeout config -> default) instead of inlining a
third hand-rolled copy of that precedence chain. Also fixes a subtle
divergence: the inline copy used os.environ.get() so a set-but-empty env var
was treated as a value rather than falling through to config; the shared
resolver .strip()s and falls through correctly.
- Added gateway.status.parse_active_agents() and routed BOTH HTTP surfaces
(/api/status and /health/detailed) through it, so the exposed active_agents
field is consistently clamped non-negative. Previously /api/status clamped
while /health/detailed exposed the raw file value, diverging on a corrupt
count.
- Added TestParseActiveAgents covering the shared coercion contract.
