Widens the symlinks=True fix to the create_profile clone sites so a
symlink pointing at a parent directory can't recurse infinitely during
'hermes profile create <name> --clone-all' (#11560). Export paths were
covered by the salvaged #58397/#58445 commits; this carries the clone
half of open PR #11573.
Fixes#11560
`hermes profile export default` crashed with `shutil.Error` when
HERMES_HOME pointed outside ~/.hermes (common in Docker deployments)
and the workspace contained broken symlinks. Two root causes:
1. `copytree` defaults to `symlinks=False` and follows link targets;
broken ones crash. #58397 (liuhao1024) drafted a minimal
`symlinks=True` flag fix; this PR adopts that change.
2. `copytree` was invoked against the entire HERMES_HOME root (which
doubles as cwd in Docker layouts). The post-hoc blacklist at
`_DEFAULT_EXPORT_EXCLUDE_ROOT` is a fixed-length enumerate-and-pray
list that can't anticipate every unrelated sibling directory
(`x11-dev/`, etc.). Replaced with a positive allow-list at
`_DEFAULT_EXPORT_INCLUDE_ROOT` enumerating the known Hermes profile
artifacts (config, persona, skills, cron, scripts, sessions,
plugins, memories, knowledge, preferences). Sensitive runtime
surfaces (`state.db`, `logs/`, auth files, other profiles) are
intentionally not in the allow-list so the export stays a
portable, credential-free snapshot of the user-facing surface —
which means the existing `test_export_default_excludes_infrastructure`
regressions remain green.
Adds two regression tests:
* test_export_default_uses_allowlist_for_unrelated_dirs — >x11-dev<
sibling directories must not leak into the archive.
* test_export_default_handles_broken_symlinks — symlinks inside
allowed artifacts survive instead of crashing the export.
closing that PR as superseded once this lands.
Closes#58394
shutil.copytree() defaults to symlinks=False which follows symlinks and
crashes on broken ones. In Docker/custom HERMES_HOME deployments,
unrelated directories may contain stale symlinks that break export.
Add symlinks=True to both copytree() calls in export_profile() so
broken symlinks are preserved as symlink entries in the archive.
Fixes#58394
video_analyze_tool's local-path branch read raw bytes via
_detect_video_mime_type (extension-only, no magic-byte check) with no
call to agent.file_safety.raise_if_read_blocked, unlike the image-gen
and video-gen provider plugins that already route local inputs through
that shared chokepoint (#57698). A model could point video_url at a
credential store (e.g. .env, auth.json) renamed or symlinked to a
video-like extension and have its raw bytes base64-encoded and sent to
the vision provider.
vision_analyze_tool and its native fast path (_vision_analyze_native)
had the same gap in their local-file branches; they were only
incidentally protected by the image magic-byte sniff rejecting
non-image content, not by the intended read guard.
Add raise_if_read_blocked() to all three local-file branches, mirroring
the existing plugins/image_gen and plugins/video_gen call sites.
* fix(cli): set correct x-initiator header per Copilot turn
copilot_default_headers() always hardcoded x-initiator: agent, but
GitHub Copilot billing requires "user" for user-initiated prompts and
"agent" for tool/follow-up calls. This caused premium requests to never
be consumed correctly, risking billing issues or account bans.
Adds is_agent_turn param to copilot_default_headers() and injects
extra_headers={"x-initiator": "user"} on the first API call of each
user turn when targeting Copilot URLs. The flag flips to False after
injection so subsequent calls (tool use, streaming fallback) default
back to "agent".
Fixes#3040
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore(release): add AUTHOR_MAP entry for @tjp2021 (PR #4097 salvage)
---------
Co-authored-by: Tim <tim@iteachyouai.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Both gateway compression entry points (session-hygiene auto-compress in
run.py; manual /compress in slash_commands.py) filtered the transcript
to user/assistant-only, content-bearing messages before calling
_compress_context. That starved the compressor:
- tool results are usually the bulk of the context, and
_prune_old_tool_results never saw them
- short filtered histories tripped the protect-first/last early-return,
so compression became a no-op even on huge sessions
- assistant tool_calls stubs (content=None) were dropped, so even the
summary lost the tool activity
Pass user/assistant/tool messages through intact, matching what the
agent loop itself feeds _compress_context.
Port of PR #3854 onto current main (the manual-compress handler moved
from run.py to slash_commands.py since the PR branched); regression test
asserts tool messages reach the compressor.
Authored-by: David Zhang <david.d.zhang@gmail.com> (@Git-on-my-level)
Co-authored-by: David Zhang <david.d.zhang@gmail.com>
The Cloud setup wizard and docs tell operators to set
WHATSAPP_CLOUD_ALLOWED_USERS (and WHATSAPP_CLOUD_ALLOW_ALL_USERS), but the
adapter DM intake gate only read WHATSAPP_CLOUD_ALLOW_FROM + WHATSAPP_CLOUD_DM_POLICY
(default open, opted-in only via GATEWAY_/WHATSAPP_ALLOW_ALL_USERS). So an
allowlist set via the documented var silently dropped every inbound
(_should_process_message -> None -> HTTP 200, no dispatch, no log line).
- _allow_from also reads WHATSAPP_CLOUD_ALLOWED_USERS
- dm_policy defaults to allowlist when an allowlist is present (else open)
- _open_dm_opted_in() also honors WHATSAPP_CLOUD_ALLOW_ALL_USERS
Explicit DM_POLICY / ALLOW_FROM still win -> backward compatible.
_do_reconnect() succeeded but never called
YuanbaoAdapter.set_active(adapter), leaving get_active()
permanently returning None after any WS disconnect/reconnect
cycle. This caused cron delivery to silently fail because
_send_yuanbao() checks get_active_adapter() and gives up
immediately when it returns None.
Fix: call set_active(adapter) after successful reconnect,
matching the pattern in connect().
Fixes#58363
Post-#48648, oversized mid-stream edits truncate to a 4096-char preview
instead of splitting. But when rich messages raise the consumer's overflow
budget to 32k, the consumer keeps accumulating past 4096 and keeps issuing
progressive edits every edit_interval — each one truncating to the SAME
preview text. Telegram counts every one of those no-op requests against the
flood budget: a long streamed reply fires ~1 identical edit per 0.8s for
the rest of the stream, trips flood control (200s+ penalties), and the
final delivery hangs behind inline flood sleeps. Users see the bot stuck
'streaming' and the chat unresponsive.
Fix at the chokepoint: track the last truncated preview per
(chat_id, message_id) and skip the API call when the new truncation is
identical. Previews still update when the visible prefix actually changes
(e.g. chunk-count marker 1/2 → 1/3). State clears on finalize and when
content shrinks back under the cap, so dedup can never mask a real edit.
Live repro: 19,956-char streamed reply, transport=edit, rich available —
4x flood-control hits within ~700ms, 250s penalties, hung final delivery.
E2E harness on the same stream: 14 edit calls on main vs 7 with the fix
(the delta is pure no-op duplicates; scales with stream length).
Self-review (3-agent + codex) findings on the async QueueListener change:
1. (HIGH) The os._exit shutdown backstop called flush_log_queue(), whose
stop() joins the listener thread unbounded. If that thread is wedged on
the rotation lock — the exact failure this change survives — shutdown
re-freezes. Add drain_log_queue(timeout): stop-only, bounded via a
throwaway joiner thread. Also release PID/runtime locks BEFORE the drain
so a slow drain can't strand them.
2. (MED) _log_queue/_queue_listener/_queued_file_handlers were read-modify-
written without a lock across register/stop/flush/reset; a gateway-init
race with a plugin/CLI path could leave two live listeners. Guard all
four globals with a single _queue_state_lock.
3. (MED) _NonFormattingQueueHandler.prepare() enqueued the same LogRecord a
synchronous handler on the emitting thread may still format/mutate.
Return copy.copy(record) (preserves msg/args/exc_info for deferred
RedactingFormatter) to remove the cross-thread mutation race.
E2E-verified: bounded drain returns in ~500ms on a permanently-wedged
listener; 4x20 concurrent flushes single-listener no-crash; args still
format and secrets still redact through the copied record.
The QueueListener change routes rotating file handlers through an
in-memory queue drained on a dedicated thread, with an atexit hook to
flush on shutdown. But _exit_after_graceful_shutdown() uses os._exit,
which bypasses atexit — so on the early-exit and #53107 hard-exit paths
the queued records (including the shutdown reason) were silently lost.
Explicitly flush_log_queue() before os._exit, and correct the now-stale
comment that claimed handlers are synchronous with nothing pending.
On Windows every Hermes process (gateway, serve, TUI/slash workers, MCP
servers, CLI commands) writes the shared rotating logs through
concurrent-log-handler's cross-process rotation lock. When the emitting
thread is an asyncio event loop, a lock wait blocks the loop — stalling it
for seconds and dropping WebSocket clients (the 'gateway keeps going down'
symptom seen in #58265).
Route every file handler through a single QueueListener on a dedicated
thread: loggers only enqueue (non-blocking); the listener does the file I/O
and rotation-lock wait off the hot path. The QueueHandler funnels via the
root logger; per-handler levels and component filters are preserved by
respect_handler_level + handler.handle on the listener thread. An atexit
hook stops the listener before logging.shutdown closes the file handlers.
- _NonFormattingQueueHandler passes the raw record (in-process queue) so
target handlers apply their own RedactingFormatter/filters.
- flush_log_queue() drains synchronously (shutdown + tests).
- rotating_file_handlers() exposes the handlers now behind the listener;
tests updated to use it.
Extends the #58265 fix: the provider-key warn-storm was one amplifier of
this contention; this takes the contention off the event loop entirely.
_normalize_custom_provider_entry() runs on every load_picker_context()
call (per picker/inventory request) and warned each time for (a) the
redundant `provider` key that Hermes' own config writer emits into
provider entries and (b) any other unknown key. On Windows the serve
launcher+worker pair share one rotating log via concurrent-log-handler's
cross-process lock, so that per-load warning volume drove 'Cannot acquire
lock after 20 attempts' retries that pegged a core, stalled the event
loop ~14s, and dropped every desktop/TUI WebSocket while /health stayed
green (gateway looked down; dashboard looked fine).
- Accept `provider` as a known key (silently ignored) so self-written
legacy configs don't warn.
- Deduplicate the normalizer's warnings per (provider, signature) so a
static config quirk is surfaced once, not on every inventory load.
Adds regression tests for both.
Fixes#58265
After a config change (e.g. switching model provider), the /new command
must clear the per-session _last_resolved_model cache so the next turn
resolves the model from the updated config instead of falling back to
the stale cached value.
Without this fix, if a transient config-cache miss occurs on the first
post-/new turn, the #35314 recovery path serves the old model from the
cache — the user sees the old model being used even though they changed
config.yaml and explicitly ran /new.
Fix applies to both call sites that reset session model state:
- GatewaySlashCommandsMixin._handle_reset_command (slash_commands.py)
- GatewayRunner compression-exhausted auto-reset (run.py)
Fixes#58403
Per-session /model overrides supplied api_key and provider but omitted
credential_pool, so billing rotation never ran on HTTP 402. Wire the pool
on fast override, rehydrate, and apply paths; backfill from provider for
legacy persisted overrides. Regression tests in tests/gateway/.
- package-lock.json changes in #58451 were unrelated peer-flag churn
- CANONICAL_PROVIDERS 'poolside' entry from #58374 has no ProviderConfig
in hermes_cli/auth.py and no setup flow, so the picker entry would be
dead; the wire-format coercions stand on their own
When OpenRouter routes to an endpoint that does not support tool/function
calling, it returns HTTP 404 with the message 'No endpoints found that
support tool use. Try disabling "browser_back".'
The raw error body does not contain 'model not found' or any other
_MODEL_NOT_FOUND_PATTERNS entry, so it falls through to FailoverReason.unknown
with retryable=True. The retry loop wastes 3-5 attempts on the same
deterministic rejection, then surfaces a confusing generic error instead of
automatically failing over to a fallback model or provider.
Adding the OpenRouter phrase to _MODEL_NOT_FOUND_PATTERNS classifies it as
model_not_found (retryable=False, should_fallback=True), which triggers the
client-error fast-fallback path in conversation_loop.py: the agent switches
to a configured fallback model/provider before the user sees the error.
Existing buffered guidance in conversation_loop.py (the 'support tool use'
hint at line ~2967) remains intact and surfaces only if every fallback
exhausts.
_try_anthropic() hard-failed (return None, None) when the anthropic
credential pool was present but had no selectable entry — e.g. the pooled
OAuth token expired and its refresh_token had gone stale, so
_select_pool_entry("anthropic") returned (True, None). This wedged every
auxiliary task routed to Anthropic (goal judge surfaced "no auxiliary
client configured") even when a perfectly valid ANTHROPIC_TOKEN /
credentials-file token was available. The main session stayed healthy
because it resolves the env token directly.
The openrouter path (_try_openrouter) and codex path already fall through
to their standalone credential on (True, None); anthropic was the only
provider that hard-failed. Make _try_anthropic fall through to
resolve_anthropic_token() on that branch so the three paths are symmetric:
a temporarily dead pool entry must not block auxiliary tasks when a valid
standalone credential exists.
Adds a regression test covering: (1) pool present + no entry + valid env
token -> client built from the env token, (2) pool present + no entry + no
resolvable token -> clean (None, None), (3) base_url defaults correctly
when falling through with pool_present=True.
api_server already caps every read via client_max_size (chunked
included), but when the limit tripped mid-read the handler's broad JSON
except turned it into 400 'Invalid JSON'. Catch
HTTPRequestEntityTooLarge in body_limit_middleware and return the
OpenAI-style 413.
Status-code polish extracted from PR #3949 by @Gutslabs — the PR's core
client_max_size change already exists on main.
The webhook adapter enforced max_body_bytes only via the Content-Length
header; a Transfer-Encoding: chunked request (content_length=None) or a
spoofed small Content-Length bypassed the cap entirely and read the full
body (bounded only by aiohttp's implicit 1 MiB default, above any
operator-configured smaller limit).
- web.Application(client_max_size=max_body_bytes): aiohttp enforces the
cap on every read path, chunked included
- catch HTTPRequestEntityTooLarge -> 413 (was swallowed into generic 400)
- post-read length re-check as defense in depth
- chunked-upload regression test
Manual port of PR #3955 by @Gutslabs onto current main (handler had
been restructured since); authorship preserved.
The container exec-read piped the whole file through base64 with no size
guard — the 50MB cap was only enforced host-side AFTER the full payload
had already streamed into host memory. A prompt-injected read of a huge
container file (or /dev/zero) could balloon the gateway process.
head -c (cap+1) bounds the read inside the sandbox; the +1 byte lets the
host distinguish at-cap from over-cap and reject with SourceTooLarge.
Input redirect replaces 'base64 --' (no argv exposure at all for
leading-dash paths). Docker integration tests re-verified live.
The salvaged #52688 rasterizer shell-out predates the TUI subprocess
stdin= guard; a rasterizer that prompts on stdin could hang the tool
under prompt_toolkit. DEVNULL it.
- _normalize_to_supported_image: ensure cache/vision exists before writing
the converted PNG (fresh HERMES_HOME had no dir -> FileNotFoundError).
- resolver: SVG passes through as image/svg+xml instead of erroring; the
call sites rasterize to PNG via the salvaged normalize step (cairosvg /
svglib / rsvg-convert / inkscape, best-effort with actionable error).
- normalization offloaded via asyncio.to_thread at both call sites.
- tests: resolver pass-through + rasterization + no-converter error paths.
- AUTHOR_MAP: jonathan@mintrx.com -> JAlmanzarMint (PR #52688 salvage).
vision_analyze embedded SVG (and BMP/TIFF) tool-results into conversation
history with media_type image/svg+xml. Anthropic only accepts jpeg/png/
gif/webp, so the request fails with a non-retryable 400. Because the image
is baked into immutable history and re-sent every turn, the session is
permanently wedged on resume — retries re-send the same bad bytes.
Add _normalize_to_supported_image(): SVG is rasterized to PNG (best-effort
via cairosvg/svglib/rsvg-convert/inkscape), other non-supported raster
formats are re-encoded to PNG via Pillow, and if conversion is impossible
the tool returns an actionable error instead of a session-wedging payload.
Wired into both the native-vision fast path and the auxiliary-API path so
the whole bug class is covered, not just the one call site.
All 99 existing vision tests pass.
- resolve_image_source: bare cwd-relative filenames ('pic.png') resolve
again (main accepted them; the path-shape gate regressed them — review
by egilewski). Unknown explicit schemes (ftp://, s3://) still rejected.
- Local backend: nonexistent path now raises a clean 'image file not
found' instead of a misleading sandbox-fallback message.
- W4: remove the now-dead path-based _detect_image_mime_type (suffix-trust
SVG acceptance) so future callers can't reintroduce it.
- W3: SVG sources rejected with a dedicated actionable message + test.
- Polish: host read_bytes / temp-file write_bytes offloaded via
asyncio.to_thread (matches the container exec-read); unused
ResolveContext.cfg/extra_roots fields dropped; duplicate policy check
documented as intentional pre-flight short-circuit.
Salvage of #35362, evolved to also close the vision sandbox-escape
(GHSA-gpxw-6wxv-w3qq). The two were the same root cause — vision read image
bytes host-side while every other tool reads through the terminal backend —
so one resolver fixes both the delivery gaps and the escape.
Delivery (from #35362, re-authored against current main since the branch was
4140 commits stale and vision_tools.py had been rewritten on both sides):
- tools/image_source.py: one resolver for data:/http(s)/file/local/container
image sources, returning raw bytes through a single magic-byte-sniff +
50MB-ingest chokepoint. Fixes 'no image attached' / 'Invalid image source'
for every source type (#7571, #25118, #29643, #22328, #32709, #9077).
- tools/credential_files.py: from_agent_visible_cache_path, the container->host
cache reverse-map (inverse of the existing forward twin).
- tools/vision_tools.py: both vision sites route through the resolver with
task_id threaded from the handler; resolved bytes are materialized to a temp
file so main's evolved encode/resize/embed-cap pipeline is reused verbatim
(kept over the PR's older bytes-core resize to avoid touching browser_tool /
conversation_compression callers).
Security (fills #35362's deliberately-stubbed _within_allowed_roots seam):
- Under a non-local terminal backend the file tools are confined to the sandbox
(SECURITY.md 2.2), but vision read host-side — a prompt-injected
vision_analyze('/etc/passwd') exfiltrated host secrets, and read_file even
redirects the model to vision_analyze for image paths. The resolver now
enforces the same boundary: local backend reads any host path (chosen
posture); non-local backend host-reads ONLY the media caches under
HERMES_HOME (where the gateway/download media lives) and routes every other
path to an in-sandbox base64 exec-read — which reads the CONTAINER's file,
the same one 'cat' would, never the host's. Paths are resolve()-d so a
symlink can't escape a cache; fail-closed when no sandbox env exists.
This closes the escape AND delivers container-only images (#32709) with the
same mechanism.
Tests: unified resolver + confinement model (tests/tools/test_image_source.py,
incl. proof a non-cache host path under Docker yields container bytes not the
host secret); existing vision tests updated to the resolver boundary; Docker
integration test verified green against a real daemon (exec-read of a tmpfs
/workspace file, a root-owned mode-600 file, and the host-secret invariant).
Fixes GHSA-gpxw-6wxv-w3qq.
Co-authored-by: banditburai <promptsiren@gmail.com>
'KEY=os.getenv(...)' / 'os.environ[...]' / 'process.env.X' values are
variable-name references in code snippets, not leaked secrets. Masking
them corrupted pasted code in prose/log contexts (issue #2852):
ha_token=os.getenv('HOMEASSISTANT_TOKEN') -> ha_token=os.get...EN').
Skip these values inside _redact_env, which covers all three passes that
share the closure (_ENV_ASSIGN_RE, _CFG_DOTTED_RE, _CFG_ANCHORED_RE).
Real secret values are still masked.
Salvage of PR #2852-fix #2854 — the PR's own placement (an unconditional
pass before the code_file gate) would have reintroduced the code-file
false-positive class; the skip is applied inside the existing gated pass
instead. Tests adapted from the PR.
Co-authored-by: crazywriter1 <sampiyonyus@gmail.com>
The Contents-API fallback's directory listing used a raw httpx.get with
no retry, so a 429/403 rate limit aborted the whole skill download even
though file fetches already retry via _github_get. Route the directory
listing through the same helper (429/reset-aware backoff, 5xx retry,
rate-limit flagging).
Salvage of PR #3033's intent — rerouted through the _github_get helper
that landed after the PR was opened, instead of the PR's ad-hoc retry
loops.
Co-authored-by: 0xbyt4 <35742124+0xbyt4@users.noreply.github.com>
The cache used a '__default__' sentinel as its path key, so switching
HERMES_HOME (profiles, tests) within one process kept serving the stale
policy loaded from the previous home. Key the cache on the actual
resolved default config path instead, so a home/config-path change
naturally misses the cache.
Trimmed from bundled PR #3923 (the other sub-fixes are superseded on
main); authored by @aydnOktay.
