mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-18 04:41:56 +00:00
feat(security): supply-chain advisory checker + lazy-install framework + tiered install fallback (#24220)
* feat(security): supply-chain advisory checker + lazy-install framework + tiered install fallback
Three coordinated mitigations for the Mini Shai-Hulud worm hitting
mistralai 2.4.6 on PyPI (2026-05-12) and for the next single-package
compromise that follows.
# What this PR makes true
1. Users with the poisoned mistralai 2.4.6 in their venv get a loud
detection banner with copy-pasteable remediation steps the moment
they run hermes (and on every gateway startup).
2. One quarantined / yanked PyPI package can no longer silently demote
a fresh install to 'core only' — the installer keeps every other
extra and tells the user which tier landed.
3. Future opt-in backends (Mistral, ElevenLabs, Honcho, etc.) can
lazy-install on first use under a strict allowlist, instead of
eagerly pulling everything at install time.
# Detection: hermes_cli/security_advisories.py
- ADVISORIES catalog (one entry currently: shai-hulud-2026-05 for
mistralai==2.4.6). Adding the next one is a single dataclass.
- detect_compromised() uses importlib.metadata.version() — no pip
dependency, works in uv venvs that lack pip.
- Banner cache (~/.hermes/cache/advisory_banner_seen) rate-limits
the startup banner to once per 24h per advisory.
- Acks persisted to security.acked_advisories in config.yaml; never
re-banner after ack.
- Wired into:
* hermes doctor — runs first, prints full remediation block
* hermes doctor --ack <id> — dismisses an advisory
* cli.py interactive run() and single-query branches — short
stderr banner pointing at hermes doctor
* gateway/run.py startup — operator-visible warning in gateway.log
# Lazy-install framework: tools/lazy_deps.py
- LAZY_DEPS allowlist maps namespaced feature keys (tts.elevenlabs,
memory.honcho, provider.bedrock, etc.) to pip specs.
- ensure(feature) installs missing deps in the active venv via the
uv → pip → ensurepip ladder (matches tools_config._pip_install).
- Strict spec safety regex rejects URLs, file paths, shell metas,
pip flag injection, control chars — only PyPI-by-name accepted.
- Gated on security.allow_lazy_installs (default true) plus the
HERMES_DISABLE_LAZY_INSTALLS env var for restricted/audited envs.
- Migrated three backends as proof of pattern:
* tools/tts_tool.py — _import_elevenlabs() calls ensure first
* plugins/memory/honcho/client.py — get_honcho_client lazy-installs
* tts.mistral / stt.mistral entries pre-registered for when PyPI
restores mistralai
# Installer fallback tiers
scripts/install.sh, scripts/install.ps1, setup-hermes.sh:
- Centralised _BROKEN_EXTRAS list (currently: mistral). Edit one
array when a transitive breaks; users keep every other extra.
- New 'all minus known-broken' tier between [all] and the existing
PyPI-only-extras tier. Only kicks in when [all] fails resolve.
- All three tiers explicit: every fallback announces which tier
landed and prints a re-run hint when not on Tier 1.
- install.ps1 and install.sh both regenerate their tier specs from
the same _BROKEN_EXTRAS array so updates stay in sync.
Side effect: install.ps1 Tier 2 spec previously hardcoded 'mistral'
in its extra list — bug fixed by the refactor (mistral is filtered
out).
# Config
hermes_cli/config.py — DEFAULT_CONFIG.security gains:
- acked_advisories: [] (advisory IDs the user has dismissed)
- allow_lazy_installs: True (security gate for ensure())
No config version bump needed — both keys nest under existing
security: block, and load_config's deep-merge picks up DEFAULT_CONFIG
defaults for users with older configs.
# Tests
tests/hermes_cli/test_security_advisories.py — 23 tests covering:
- detect_compromised matches/non-matches, wildcard frozenset
- ack persistence, idempotence, blank rejection, config-failure path
- banner cache rate limiting + 24h re-banner + ack-stops-banner
- short_banner_lines / full_remediation_text / render_doctor_section /
gateway_log_message
- shipped catalog well-formedness invariant
tests/tools/test_lazy_deps.py — 40 tests covering:
- spec safety: 11 safe parametrized + 18 unsafe parametrized
- allowlist: unknown-feature rejection, namespace.name shape,
every shipped spec passes the safety regex
- security gating: config flag, env var, default, fail-open
- ensure() happy/sad paths: already-satisfied, install success,
pip stderr surfaced on failure, install-succeeds-but-still-missing
- is_available, feature_install_command
Combined: 63 new tests, all passing under scripts/run_tests.sh.
# Validation
- scripts/run_tests.sh tests/hermes_cli/test_security_advisories.py
tests/tools/test_lazy_deps.py → 63/63 passing
- scripts/run_tests.sh tests/hermes_cli/test_doctor.py
tests/hermes_cli/test_doctor_command_install.py
tests/tools/test_tts_mistral.py tests/tools/test_transcription_tools.py
tests/tools/test_transcription_dotenv_fallback.py → 165/165 passing
- scripts/run_tests.sh tests/hermes_cli/ tests/tools/ →
9191 passed, 8 pre-existing failures (verified on origin/main
before this change)
- bash -n on install.sh and setup-hermes.sh → OK
- py_compile on all modified .py files → OK
- End-to-end smoke test of detect_compromised + render_doctor_section
+ gateway_log_message with mocked installed version → produces
copy-pasteable remediation output
# Community
Full advisory + remediation steps:
website/docs/community/security-advisories/shai-hulud-mistralai-2026-05.md
Short-form post drafts (Discord, GitHub pinned issue, README banner):
scripts/community-announcement-shai-hulud.md
Refs: PR #24205 (mistral disabled), Socket Security advisory
<https://socket.dev/blog/mini-shai-hulud-worm-pypi>
* build(deps): pin every direct dep to ==X.Y.Z (no ranges)
Companion to the supply-chain advisory work: replace every >=/</~= range
in pyproject.toml's [project.dependencies] and [project.optional-dependencies]
with an exact ==X.Y.Z pin sourced from uv.lock.
Why: ranges allow PyPI to ship a fresh version of any direct dep at any
time without a code review on our side. With ranges, the malicious
mistralai 2.4.6 release would have been pulled by every fresh
'pip install -e .[all]' for the hours between upload and PyPI's
quarantine — exactly the install window we got hit on. Exact pins close
that window: the only way a new package version reaches a user is via
an intentional update on our end.
What the user-facing change is: nothing, behavior-wise. Every package
resolves to the same version it was already resolving to via uv.lock —
the pins just remove the resolver's freedom to pick a different one.
Cost: any user installing Hermes alongside another package that requires
a newer pin gets a resolver conflict. Acceptable for our isolated-venv
install path; documented in the new comment block.
Build-system requires line (setuptools>=61.0) is intentionally left
as a range — pinning the build backend would block fresh pip from
bootstrapping the build on architectures where that exact wheel isn't
available.
mistral extra (mistralai==2.3.0) is pinned but stays out of [all]
(per PR #24205). 'uv lock' regeneration will fail until PyPI restores
mistralai; lockfile regeneration is gated behind that, NOT on every PR.
LAZY_DEPS in tools/lazy_deps.py also moved to exact pins so the lazy-
install pathway can never resolve a different version than the one
declared in pyproject.toml.
Validation:
- Cross-checked all 77 pinned direct deps in pyproject.toml against
uv.lock — every pin matches the resolved version exactly.
- Cross-checked all LAZY_DEPS specs against uv.lock — same.
- 'uv pip install -e .[all] --dry-run' resolves 205 packages cleanly.
- tests/tools/test_lazy_deps.py + tests/hermes_cli/test_security_advisories.py
→ 63/63 passing (every shipped spec passes the safety regex).
- Doctor + TTS + transcription targeted suite → 146/146 passing.
* build(deps): hash-verify transitives via uv.lock; remove unresolvable [mistral] extra
You asked: 'what about the dependencies the dependencies rely on?' —
correctly noting that exact-pinning direct deps in pyproject.toml does
NOT cover the transitive graph. `pip install` and `uv pip install` both
re-resolve transitives fresh from PyPI at install time, so a compromised
transitive (e.g. `httpcore` if it got worm-poisoned tomorrow) would
still hit our users even with every direct dep exact-pinned.
# What this commit fixes
1. **Both real installer scripts now prefer `uv sync --locked` as Tier 0.**
uv.lock records SHA256 hashes for every transitive — a compromised
package with a different hash gets REJECTED. Falls through to the
existing `uv pip install` cascade if the lockfile is missing or
stale, with a loud warning that the fallback path does NOT
hash-verify transitives. Previously only `setup-hermes.sh` (the dev
path) used the lockfile; `scripts/install.sh` and `scripts/install.ps1`
(the paths fresh users actually run) skipped it.
2. **Removed the `[mistral]` extra entirely.** The `mistralai` PyPI
project is fully quarantined right now — every version returns 404,
so any pin we wrote was unresolvable, which broke `uv lock --check`
in CI. Restoration is documented in pyproject.toml as a 5-step
checklist (verify, re-add extra, re-enable in 4 modules, regenerate
lock, optionally re-add to [all]).
3. **Regenerated uv.lock.** 262 packages, mistralai/eval-type-backport/
jsonpath-python pruned. `uv lock --check` now passes.
# Defense-in-depth view
| Layer | Where | Protects against |
|----------------------------|-------------------|-------------------------------------------|
| Exact pins in pyproject | direct deps | new mistralai 2.4.6-style direct compromise |
| uv.lock + `--locked` install | transitive graph | transitive worm injection |
| Tier-0 hash-verified path | install.sh / .ps1 | actually USE the lockfile in fresh installs |
| `uv lock --check` CI gate | every PR | drift between pyproject and lockfile |
| `hermes_cli/security_advisories.py` | runtime | cleanup for users who already got hit |
The exact pinning + hash verification together close the supply-chain
gap. Without the lockfile path, exact pins alone are theater.
# Validation
- `uv lock --check` → passes (262 packages resolved, no drift).
- `bash -n` on install.sh + setup-hermes.sh → OK.
- 209/209 tests passing across new + adjacent test files
(test_lazy_deps.py, test_security_advisories.py, test_doctor.py,
test_tts_mistral.py, test_transcription_tools.py).
- TOML parse OK.
* chore: remove community announcement drafts (PR body covers it)
* build(deps): lazy-install every opt-in backend (anthropic, search, terminal, platforms, dashboard)
Extends the lazy-install framework to cover everything that's not used by
every hermes session. Base install drops from ~60 packages to 45.
Moved out of core dependencies = []:
- anthropic (only when provider=anthropic native, not via aggregators)
- exa-py, firecrawl-py, parallel-web (search backends; only when picked)
- fal-client (image gen; only when picked)
- edge-tts (default TTS but still optional)
New extras in pyproject.toml: [anthropic] [exa] [firecrawl] [parallel-web]
[fal] [edge-tts]. All added to [all].
New LAZY_DEPS entries: provider.anthropic, search.{exa,firecrawl,parallel},
tts.edge, image.fal, memory.hindsight, platform.{telegram,discord,matrix},
terminal.{modal,daytona,vercel}, tool.dashboard.
Each import site now calls ensure() before importing the SDK. Where the
module had a top-level try/except (telegram, discord, fastapi), the
graceful-fallback pattern was extended to lazy-install on first
check_*_requirements() call and re-bind module globals.
Updated test_windows_native_support.py tzdata check from snapshot
(>=2023.3 literal) to invariant (any version + win32 marker).
Validation:
- Base install: 45 packages (was ~60); 6 newly-extracted packages absent
- uv lock --check: passes (262 packages, no drift)
- 209/209 lazy_deps + advisory + doctor + tts/transcription tests passing
- py_compile clean on all 12 modified modules
This commit is contained in:
Teknium
GitHub
parent
99ad2d1372
commit
c1eb2dcda7
28 changed files with 2433 additions and 243 deletions
|
|
@ -51,6 +51,13 @@ class DaytonaEnvironment(BaseEnvironment):
|
|||
requested_cwd = cwd
|
||||
super().__init__(cwd=cwd, timeout=timeout)
|
||||
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("terminal.daytona", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
from daytona import (
|
||||
Daytona,
|
||||
CreateSandboxFromImageParams,
|
||||
|
|
|
|||
|
|
@ -80,11 +80,23 @@ def _delete_direct_snapshot(task_id: str, snapshot_id: str | None = None) -> Non
|
|||
_save_snapshots(snapshots)
|
||||
|
||||
|
||||
def _ensure_modal_sdk() -> None:
|
||||
"""Lazy-install modal on demand. Idempotent — fast no-op once installed."""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("terminal.modal", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
|
||||
|
||||
def _resolve_modal_image(image_spec: Any) -> Any:
|
||||
"""Convert registry references or snapshot ids into Modal image objects.
|
||||
|
||||
Includes add_python support for ubuntu/debian images (absorbed from PR 4511).
|
||||
"""
|
||||
_ensure_modal_sdk()
|
||||
import modal as _modal
|
||||
|
||||
if not isinstance(image_spec, str):
|
||||
|
|
@ -183,6 +195,7 @@ class ModalEnvironment(BaseEnvironment):
|
|||
if restored_snapshot_id:
|
||||
logger.info("Modal: restoring from snapshot %s", restored_snapshot_id[:20])
|
||||
|
||||
_ensure_modal_sdk()
|
||||
import modal as _modal
|
||||
|
||||
cred_mounts = []
|
||||
|
|
|
|||
|
|
@ -42,6 +42,19 @@ if TYPE_CHECKING:
|
|||
|
||||
DEFAULT_VERCEL_CWD = "/vercel/sandbox"
|
||||
_DEFAULT_CONTAINER_DISK_MB = 51200
|
||||
|
||||
|
||||
def _ensure_vercel_sdk() -> None:
|
||||
"""Lazy-install vercel SDK on demand. Idempotent."""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("terminal.vercel", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
|
||||
|
||||
_CREATE_RETRY_ATTEMPTS = 3
|
||||
_WRITE_RETRY_ATTEMPTS = 3
|
||||
_TRANSIENT_STATUS_CODES = frozenset({408, 425, 429, 500, 502, 503, 504})
|
||||
|
|
@ -194,6 +207,7 @@ def _extract_snapshot_id(snapshot: Any) -> str | None:
|
|||
|
||||
@cache
|
||||
def _sandbox_status_type() -> type[SandboxStatus]:
|
||||
_ensure_vercel_sdk()
|
||||
from vercel.sandbox import SandboxStatus
|
||||
|
||||
return SandboxStatus
|
||||
|
|
@ -260,6 +274,7 @@ class VercelSandboxEnvironment(BaseEnvironment):
|
|||
"Use the default shared setting."
|
||||
)
|
||||
|
||||
_ensure_vercel_sdk()
|
||||
from vercel.sandbox import Resources
|
||||
|
||||
sandbox_timeout = max(
|
||||
|
|
@ -281,6 +296,7 @@ class VercelSandboxEnvironment(BaseEnvironment):
|
|||
)
|
||||
|
||||
def _create_sandbox(self) -> Sandbox:
|
||||
_ensure_vercel_sdk()
|
||||
from vercel.sandbox import Sandbox
|
||||
|
||||
snapshot_id = _get_snapshot_id(self._task_id) if self._persistent else None
|
||||
|
|
|
|||
|
|
@ -52,6 +52,13 @@ def _load_fal_client() -> Any:
|
|||
global fal_client
|
||||
if fal_client is not None:
|
||||
return fal_client
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("image.fal", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
import fal_client as _fal_client # noqa: F811 — module-global rebind
|
||||
fal_client = _fal_client
|
||||
return fal_client
|
||||
|
|
|
|||
441
tools/lazy_deps.py
Normal file
441
tools/lazy_deps.py
Normal file
|
|
@ -0,0 +1,441 @@
|
|||
"""
|
||||
Lazy dependency installer for opt-in Hermes Agent backends.
|
||||
|
||||
Many Hermes features (Mistral TTS, ElevenLabs TTS, Honcho memory, Bedrock,
|
||||
Slack, Matrix, etc.) require Python packages that not every user needs. The
|
||||
historical approach was to bundle them all under ``pyproject.toml`` extras
|
||||
(``hermes-agent[all]``) and install them eagerly at setup time. That has
|
||||
two problems:
|
||||
|
||||
1. **Fragility.** When one extra's transitive dependency becomes
|
||||
unavailable on PyPI (quarantined for malware, yanked, broken upload),
|
||||
the *entire* ``[all]`` resolve fails and fresh installs silently fall
|
||||
back to a stripped tier — losing 10+ unrelated extras at once.
|
||||
|
||||
2. **Bloat.** A user who only ever talks to one provider pulls hundreds
|
||||
of packages they will never import.
|
||||
|
||||
The lazy-install pattern fixes both. Backends call :func:`ensure` at the
|
||||
top of their first-import path. If the deps are missing, ``ensure`` checks
|
||||
the ``security.allow_lazy_installs`` config flag (default true) and runs
|
||||
a venv-scoped pip install. If the user has explicitly disabled lazy
|
||||
installs, ``ensure`` raises :class:`FeatureUnavailable` with a clear
|
||||
remediation hint pointing at ``hermes tools`` or the manual pip command.
|
||||
|
||||
Security model:
|
||||
|
||||
* **Venv-scoped only.** Installs target ``sys.executable`` in the active
|
||||
venv. We never touch the system Python.
|
||||
* **PyPI by package name only.** Specs may be ``"package>=1.0,<2"`` etc.
|
||||
We do NOT support ``--index-url`` overrides, ``git+https://``, file:
|
||||
paths, or any other input that could be hijacked by a malicious config.
|
||||
* **Allowlist.** Only specs that appear in :data:`LAZY_DEPS` can be
|
||||
installed via this path. A typo in feature name doesn't get the user
|
||||
install-anything semantics.
|
||||
* **Opt-out.** Setting ``security.allow_lazy_installs: false`` in
|
||||
``config.yaml`` disables runtime installs. Users in restricted networks
|
||||
or strict security postures can pin themselves to whatever was installed
|
||||
at setup time.
|
||||
* **Offline detection.** If the install fails (offline, mirror down,
|
||||
PyPI 404 / quarantine), we surface the failure as
|
||||
:class:`FeatureUnavailable` with the actual pip stderr — no silent
|
||||
retries, no caching of bad state.
|
||||
|
||||
Adding a new backend:
|
||||
|
||||
1. Add an entry to :data:`LAZY_DEPS` with the package specs.
|
||||
2. At the top of the backend module's import path, call
|
||||
``ensure("feature.name")`` inside a try/except that converts
|
||||
:class:`FeatureUnavailable` to a useful runtime error.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Allowlist of lazy-installable backends.
|
||||
#
|
||||
# Keys are dot-separated feature names ("namespace.backend"). Values are
|
||||
# tuples of pip-installable specs that match the corresponding extra in
|
||||
# pyproject.toml. The framework enforces that only specs from this map
|
||||
# can flow into the pip install command.
|
||||
# =============================================================================
|
||||
|
||||
|
||||
LAZY_DEPS: dict[str, tuple[str, ...]] = {
|
||||
# ─── Inference providers ───────────────────────────────────────────────
|
||||
# Native Anthropic SDK — needed when provider=anthropic (not via
|
||||
# OpenRouter / aggregators which use the openai SDK).
|
||||
"provider.anthropic": ("anthropic==0.86.0",),
|
||||
# AWS Bedrock provider
|
||||
"provider.bedrock": ("boto3==1.42.89",),
|
||||
|
||||
# ─── Web search backends ───────────────────────────────────────────────
|
||||
"search.exa": ("exa-py==2.10.2",),
|
||||
"search.firecrawl": ("firecrawl-py==4.17.0",),
|
||||
"search.parallel": ("parallel-web==0.4.2",),
|
||||
|
||||
# ─── TTS providers ─────────────────────────────────────────────────────
|
||||
# Pinned to exact versions to match pyproject.toml's no-ranges policy
|
||||
# (see comment at top of [project.dependencies]). When bumping, update
|
||||
# both this map AND the corresponding extra in pyproject.toml.
|
||||
#
|
||||
# NOTE: tts.mistral / stt.mistral entries are intentionally absent —
|
||||
# the `mistralai` PyPI project is quarantined as of 2026-05-12 (Mini
|
||||
# Shai-Hulud worm). Re-add when PyPI restores a clean release; see
|
||||
# comment in pyproject.toml above the (removed) `mistral` extra for
|
||||
# the full restoration checklist.
|
||||
"tts.edge": ("edge-tts==7.2.7",),
|
||||
"tts.elevenlabs": ("elevenlabs==1.59.0",),
|
||||
|
||||
# ─── Speech-to-text providers ──────────────────────────────────────────
|
||||
"stt.faster_whisper": (
|
||||
"faster-whisper==1.2.1",
|
||||
"sounddevice==0.5.5",
|
||||
"numpy==2.4.3",
|
||||
),
|
||||
|
||||
# ─── Image generation backends ─────────────────────────────────────────
|
||||
"image.fal": ("fal-client==0.13.1",),
|
||||
|
||||
# ─── Memory providers ──────────────────────────────────────────────────
|
||||
"memory.honcho": ("honcho-ai==2.0.1",),
|
||||
"memory.hindsight": ("hindsight-client==0.6.1",),
|
||||
|
||||
# ─── Messaging platforms (lazy-installable on demand) ──────────────────
|
||||
"platform.telegram": ("python-telegram-bot[webhooks]==22.6",),
|
||||
"platform.discord": ("discord.py[voice]==2.7.1",),
|
||||
"platform.slack": (
|
||||
"slack-bolt==1.27.0",
|
||||
"slack-sdk==3.40.1",
|
||||
),
|
||||
"platform.matrix": (
|
||||
"mautrix[encryption]==0.21.0",
|
||||
"Markdown==3.10.2",
|
||||
"aiosqlite==0.22.1",
|
||||
"asyncpg==0.31.0",
|
||||
"aiohttp-socks==0.11.0",
|
||||
),
|
||||
"platform.dingtalk": (
|
||||
"dingtalk-stream==0.24.3",
|
||||
"alibabacloud-dingtalk==2.2.42",
|
||||
"qrcode==7.4.2",
|
||||
),
|
||||
"platform.feishu": (
|
||||
"lark-oapi==1.5.3",
|
||||
"qrcode==7.4.2",
|
||||
),
|
||||
|
||||
# ─── Terminal backends ─────────────────────────────────────────────────
|
||||
"terminal.modal": ("modal==1.3.4",),
|
||||
"terminal.daytona": ("daytona==0.155.0",),
|
||||
"terminal.vercel": ("vercel==0.5.7",),
|
||||
|
||||
# ─── Skills ────────────────────────────────────────────────────────────
|
||||
"skill.google_workspace": (
|
||||
"google-api-python-client==2.194.0",
|
||||
"google-auth-oauthlib==1.3.1",
|
||||
"google-auth-httplib2==0.3.1",
|
||||
),
|
||||
"skill.youtube": ("youtube-transcript-api==1.2.4",),
|
||||
|
||||
# ─── Tools ─────────────────────────────────────────────────────────────
|
||||
# ACP adapter (VS Code / Zed / JetBrains integration)
|
||||
"tool.acp": ("agent-client-protocol==0.9.0",),
|
||||
# Dashboard (`hermes dashboard`)
|
||||
"tool.dashboard": (
|
||||
"fastapi==0.133.1",
|
||||
"uvicorn[standard]==0.41.0",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# Conservative regex for spec validation — package name plus optional
|
||||
# version range. Reject anything that looks like a URL, file path, or shell
|
||||
# metacharacter.
|
||||
_SAFE_SPEC = re.compile(
|
||||
r"^[A-Za-z0-9_][A-Za-z0-9_.\-]*" # package name
|
||||
r"(?:\[[A-Za-z0-9_,\-]+\])?" # optional [extras]
|
||||
r"(?:[<>=!~]=?[A-Za-z0-9_.\-+,*<>=!~]+)?" # optional version specifier
|
||||
r"$"
|
||||
)
|
||||
|
||||
|
||||
class FeatureUnavailable(RuntimeError):
|
||||
"""A lazily-installable feature is missing and cannot be made available.
|
||||
|
||||
Either the deps were never installed and the user has disabled lazy
|
||||
installs, or the install attempt failed.
|
||||
"""
|
||||
|
||||
def __init__(self, feature: str, missing: tuple[str, ...], reason: str):
|
||||
self.feature = feature
|
||||
self.missing = missing
|
||||
self.reason = reason
|
||||
super().__init__(self._format())
|
||||
|
||||
def _format(self) -> str:
|
||||
spec_list = " ".join(repr(s) for s in self.missing)
|
||||
return (
|
||||
f"Feature {self.feature!r} unavailable: {self.reason}. "
|
||||
f"To enable manually: uv pip install {spec_list} "
|
||||
f"(or: pip install {spec_list})."
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _InstallResult:
|
||||
success: bool
|
||||
stdout: str
|
||||
stderr: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Internals
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def _allow_lazy_installs() -> bool:
|
||||
"""Return the ``security.allow_lazy_installs`` config flag.
|
||||
|
||||
Defaults to True. If config is unreadable we fail open (allow), because
|
||||
refusing to install would lock people out of their own backends; the
|
||||
decision to block is an explicit user opt-in.
|
||||
"""
|
||||
if os.environ.get("HERMES_DISABLE_LAZY_INSTALLS") == "1":
|
||||
return False
|
||||
try:
|
||||
from hermes_cli.config import load_config
|
||||
cfg = load_config()
|
||||
except Exception:
|
||||
return True
|
||||
sec = cfg.get("security") or {}
|
||||
val = sec.get("allow_lazy_installs", True)
|
||||
return bool(val)
|
||||
|
||||
|
||||
def _spec_is_safe(spec: str) -> bool:
|
||||
"""Reject pip specs that contain URLs, paths, or shell metacharacters."""
|
||||
if not spec or len(spec) > 200:
|
||||
return False
|
||||
if any(ch in spec for ch in (";", "|", "&", "`", "$", "\n", "\r", "\t", "\\")):
|
||||
return False
|
||||
if spec.startswith(("-", "/", ".")) or "://" in spec or "@" in spec:
|
||||
return False
|
||||
return bool(_SAFE_SPEC.match(spec))
|
||||
|
||||
|
||||
def _pkg_name_from_spec(spec: str) -> str:
|
||||
"""Extract the bare package name from a pip spec.
|
||||
|
||||
``"slack-bolt>=1.18.0,<2"`` → ``"slack-bolt"``
|
||||
``"mautrix[encryption]>=0.20"`` → ``"mautrix"``
|
||||
"""
|
||||
m = re.match(r"^([A-Za-z0-9_][A-Za-z0-9_.\-]*)", spec)
|
||||
return m.group(1) if m else spec
|
||||
|
||||
|
||||
def _is_satisfied(spec: str) -> bool:
|
||||
"""Best-effort check: is ``spec`` already satisfied in the current env?
|
||||
|
||||
We don't enforce the version range — if the package is importable
|
||||
we assume the user knows what they're doing. This matches how the
|
||||
lazy-import sites already behave.
|
||||
"""
|
||||
pkg = _pkg_name_from_spec(spec)
|
||||
try:
|
||||
from importlib.metadata import PackageNotFoundError, version
|
||||
except ImportError:
|
||||
return False
|
||||
try:
|
||||
version(pkg)
|
||||
return True
|
||||
except PackageNotFoundError:
|
||||
return False
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def _venv_pip_install(specs: tuple[str, ...], *, timeout: int = 300) -> _InstallResult:
|
||||
"""Install ``specs`` into the active venv using uv → pip → ensurepip ladder.
|
||||
|
||||
Mirrors the strategy in ``hermes_cli.tools_config._pip_install`` but
|
||||
kept independent here so this module has no CLI dependency.
|
||||
"""
|
||||
if not specs:
|
||||
return _InstallResult(True, "", "")
|
||||
|
||||
venv_root = Path(sys.executable).parent.parent
|
||||
uv_env = {**os.environ, "VIRTUAL_ENV": str(venv_root)}
|
||||
|
||||
# Tier 1: uv (preferred — fast, doesn't need pip in the venv)
|
||||
uv_bin = shutil.which("uv")
|
||||
if uv_bin:
|
||||
try:
|
||||
r = subprocess.run(
|
||||
[uv_bin, "pip", "install", *specs],
|
||||
capture_output=True, text=True, timeout=timeout, env=uv_env,
|
||||
)
|
||||
if r.returncode == 0:
|
||||
return _InstallResult(True, r.stdout or "", r.stderr or "")
|
||||
logger.debug("uv pip install failed: %s", r.stderr)
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError) as e:
|
||||
logger.debug("uv invocation failed: %s", e)
|
||||
|
||||
# Tier 2: python -m pip (with ensurepip bootstrap if needed)
|
||||
pip_cmd = [sys.executable, "-m", "pip"]
|
||||
try:
|
||||
probe = subprocess.run(
|
||||
pip_cmd + ["--version"],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
if probe.returncode != 0:
|
||||
raise FileNotFoundError("pip not in venv")
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
try:
|
||||
subprocess.run(
|
||||
[sys.executable, "-m", "ensurepip", "--upgrade", "--default-pip"],
|
||||
capture_output=True, text=True, timeout=120, check=True,
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
|
||||
return _InstallResult(False, "",
|
||||
f"pip not available and ensurepip failed: {e}")
|
||||
|
||||
try:
|
||||
r = subprocess.run(
|
||||
pip_cmd + ["install", *specs],
|
||||
capture_output=True, text=True, timeout=timeout,
|
||||
)
|
||||
return _InstallResult(r.returncode == 0, r.stdout or "", r.stderr or "")
|
||||
except subprocess.TimeoutExpired as e:
|
||||
return _InstallResult(False, "", f"pip install timed out: {e}")
|
||||
except Exception as e:
|
||||
return _InstallResult(False, "", f"pip install failed: {e}")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Public API
|
||||
# =============================================================================
|
||||
|
||||
|
||||
def feature_specs(feature: str) -> tuple[str, ...]:
|
||||
"""Return the registered specs for a feature, or raise KeyError."""
|
||||
if feature not in LAZY_DEPS:
|
||||
raise KeyError(f"Unknown lazy feature: {feature!r}")
|
||||
return LAZY_DEPS[feature]
|
||||
|
||||
|
||||
def feature_missing(feature: str) -> tuple[str, ...]:
|
||||
"""Return the subset of specs for ``feature`` not currently installed."""
|
||||
return tuple(s for s in feature_specs(feature) if not _is_satisfied(s))
|
||||
|
||||
|
||||
def ensure(feature: str, *, prompt: bool = True) -> None:
|
||||
"""Make sure all packages for ``feature`` are importable.
|
||||
|
||||
If they're missing, attempts to install them in the active venv. Raises
|
||||
:class:`FeatureUnavailable` if the user has disabled lazy installs or
|
||||
if the install attempt fails.
|
||||
|
||||
``prompt``: when True (default) and stdin is a TTY, asks the user to
|
||||
confirm before installing. Non-interactive callers (gateway, cron,
|
||||
batch) get prompt=False and skip the confirmation — config flag is
|
||||
the gate in that case.
|
||||
"""
|
||||
if feature not in LAZY_DEPS:
|
||||
raise FeatureUnavailable(
|
||||
feature, (), f"feature {feature!r} not in LAZY_DEPS allowlist"
|
||||
)
|
||||
|
||||
missing = feature_missing(feature)
|
||||
if not missing:
|
||||
return
|
||||
|
||||
# Validate every spec against the allowlist + safety regex. Belt and
|
||||
# braces — the keys-in-LAZY_DEPS check above already constrains this.
|
||||
for spec in missing:
|
||||
if not _spec_is_safe(spec):
|
||||
raise FeatureUnavailable(
|
||||
feature, missing,
|
||||
f"refusing to install unsafe spec {spec!r}"
|
||||
)
|
||||
|
||||
if not _allow_lazy_installs():
|
||||
raise FeatureUnavailable(
|
||||
feature, missing,
|
||||
"lazy installs disabled (security.allow_lazy_installs=false)"
|
||||
)
|
||||
|
||||
if prompt and sys.stdin.isatty() and sys.stdout.isatty():
|
||||
spec_list = ", ".join(missing)
|
||||
try:
|
||||
answer = input(
|
||||
f"\nFeature {feature!r} requires: {spec_list}\n"
|
||||
f"Install into the active venv now? [Y/n] "
|
||||
).strip().lower()
|
||||
except (EOFError, KeyboardInterrupt):
|
||||
answer = "n"
|
||||
if answer and answer not in ("y", "yes"):
|
||||
raise FeatureUnavailable(
|
||||
feature, missing, "user declined install at prompt"
|
||||
)
|
||||
|
||||
logger.info("Lazy-installing %s for feature %r", " ".join(missing), feature)
|
||||
result = _venv_pip_install(missing)
|
||||
if not result.success:
|
||||
# Surface the actual pip error so the user can debug PyPI-side
|
||||
# issues (404 quarantine, network down, etc.).
|
||||
snippet = (result.stderr or result.stdout or "").strip()
|
||||
if snippet:
|
||||
# Clip to a readable size — pip can dump pages of resolution traces.
|
||||
snippet = snippet[-2000:]
|
||||
raise FeatureUnavailable(
|
||||
feature, missing,
|
||||
f"pip install failed: {snippet or 'no error output'}"
|
||||
)
|
||||
|
||||
# Verify post-install. importlib.metadata caches per-process, so if we
|
||||
# just installed something the cache may not see it without a refresh.
|
||||
try:
|
||||
import importlib.metadata as _md
|
||||
if hasattr(_md, "_cache_clear"):
|
||||
_md._cache_clear() # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
still_missing = feature_missing(feature)
|
||||
if still_missing:
|
||||
raise FeatureUnavailable(
|
||||
feature, still_missing,
|
||||
"install reported success but packages still not importable "
|
||||
"(may require Python restart)"
|
||||
)
|
||||
|
||||
logger.info("Lazy install complete for feature %r", feature)
|
||||
|
||||
|
||||
def is_available(feature: str) -> bool:
|
||||
"""Return True if the feature's deps are already satisfied."""
|
||||
if feature not in LAZY_DEPS:
|
||||
return False
|
||||
return not feature_missing(feature)
|
||||
|
||||
|
||||
def feature_install_command(feature: str) -> Optional[str]:
|
||||
"""Return the ``pip install`` command a user could run manually, or None."""
|
||||
if feature not in LAZY_DEPS:
|
||||
return None
|
||||
specs = LAZY_DEPS[feature]
|
||||
return "uv pip install " + " ".join(repr(s) for s in specs)
|
||||
|
|
@ -80,11 +80,34 @@ from tools.xai_http import hermes_xai_user_agent
|
|||
|
||||
def _import_edge_tts():
|
||||
"""Lazy import edge_tts. Returns the module or raises ImportError."""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("tts.edge", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
import edge_tts
|
||||
return edge_tts
|
||||
|
||||
def _import_elevenlabs():
|
||||
"""Lazy import ElevenLabs client. Returns the class or raises ImportError."""
|
||||
"""Lazy import ElevenLabs client. Returns the class or raises ImportError.
|
||||
|
||||
Calls :func:`tools.lazy_deps.ensure` first so the SDK gets installed on
|
||||
demand if the user picked ElevenLabs as their TTS provider but never ran
|
||||
the post-setup hook (e.g. enabled it by editing config.yaml directly).
|
||||
Raises ``ImportError`` on lazy-install failure so existing callers'
|
||||
error-handling paths keep working.
|
||||
"""
|
||||
try:
|
||||
from tools.lazy_deps import FeatureUnavailable, ensure
|
||||
ensure("tts.elevenlabs", prompt=False)
|
||||
except ImportError:
|
||||
# lazy_deps module itself missing — fall through to the raw import
|
||||
# so older code paths still get a clean ImportError.
|
||||
pass
|
||||
except Exception as e: # FeatureUnavailable or any unexpected error
|
||||
raise ImportError(str(e))
|
||||
from elevenlabs.client import ElevenLabs
|
||||
return ElevenLabs
|
||||
|
||||
|
|
|
|||
|
|
@ -64,6 +64,13 @@ def _load_firecrawl_cls() -> type:
|
|||
"""Import and cache ``firecrawl.Firecrawl``."""
|
||||
global _FIRECRAWL_CLS_CACHE
|
||||
if _FIRECRAWL_CLS_CACHE is None:
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("search.firecrawl", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
from firecrawl import Firecrawl as _cls
|
||||
_FIRECRAWL_CLS_CACHE = _cls
|
||||
return _FIRECRAWL_CLS_CACHE
|
||||
|
|
@ -358,6 +365,13 @@ def _get_parallel_client():
|
|||
|
||||
Requires PARALLEL_API_KEY environment variable.
|
||||
"""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("search.parallel", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
from parallel import Parallel
|
||||
global _parallel_client
|
||||
if _parallel_client is None:
|
||||
|
|
@ -376,6 +390,13 @@ def _get_async_parallel_client():
|
|||
|
||||
Requires PARALLEL_API_KEY environment variable.
|
||||
"""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("search.parallel", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
from parallel import AsyncParallel
|
||||
global _async_parallel_client
|
||||
if _async_parallel_client is None:
|
||||
|
|
@ -990,6 +1011,13 @@ def _get_exa_client():
|
|||
|
||||
Requires EXA_API_KEY environment variable.
|
||||
"""
|
||||
try:
|
||||
from tools.lazy_deps import ensure as _lazy_ensure
|
||||
_lazy_ensure("search.exa", prompt=False)
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
raise ImportError(str(e))
|
||||
from exa_py import Exa
|
||||
global _exa_client
|
||||
if _exa_client is None:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue