feat(security): supply-chain advisory checker + lazy-install framework + tiered install fallback (#24220)

* feat(security): supply-chain advisory checker + lazy-install framework + tiered install fallback

Three coordinated mitigations for the Mini Shai-Hulud worm hitting
mistralai 2.4.6 on PyPI (2026-05-12) and for the next single-package
compromise that follows.

# What this PR makes true

1. Users with the poisoned mistralai 2.4.6 in their venv get a loud
   detection banner with copy-pasteable remediation steps the moment
   they run hermes (and on every gateway startup).
2. One quarantined / yanked PyPI package can no longer silently demote
   a fresh install to 'core only' — the installer keeps every other
   extra and tells the user which tier landed.
3. Future opt-in backends (Mistral, ElevenLabs, Honcho, etc.) can
   lazy-install on first use under a strict allowlist, instead of
   eagerly pulling everything at install time.

# Detection: hermes_cli/security_advisories.py

- ADVISORIES catalog (one entry currently: shai-hulud-2026-05 for
  mistralai==2.4.6). Adding the next one is a single dataclass.
- detect_compromised() uses importlib.metadata.version() — no pip
  dependency, works in uv venvs that lack pip.
- Banner cache (~/.hermes/cache/advisory_banner_seen) rate-limits
  the startup banner to once per 24h per advisory.
- Acks persisted to security.acked_advisories in config.yaml; never
  re-banner after ack.
- Wired into:
  * hermes doctor — runs first, prints full remediation block
  * hermes doctor --ack <id> — dismisses an advisory
  * cli.py interactive run() and single-query branches — short
    stderr banner pointing at hermes doctor
  * gateway/run.py startup — operator-visible warning in gateway.log

# Lazy-install framework: tools/lazy_deps.py

- LAZY_DEPS allowlist maps namespaced feature keys (tts.elevenlabs,
  memory.honcho, provider.bedrock, etc.) to pip specs.
- ensure(feature) installs missing deps in the active venv via the
  uv → pip → ensurepip ladder (matches tools_config._pip_install).
- Strict spec safety regex rejects URLs, file paths, shell metas,
  pip flag injection, control chars — only PyPI-by-name accepted.
- Gated on security.allow_lazy_installs (default true) plus the
  HERMES_DISABLE_LAZY_INSTALLS env var for restricted/audited envs.
- Migrated three backends as proof of pattern:
  * tools/tts_tool.py — _import_elevenlabs() calls ensure first
  * plugins/memory/honcho/client.py — get_honcho_client lazy-installs
  * tts.mistral / stt.mistral entries pre-registered for when PyPI
    restores mistralai

# Installer fallback tiers

scripts/install.sh, scripts/install.ps1, setup-hermes.sh:

- Centralised _BROKEN_EXTRAS list (currently: mistral). Edit one
  array when a transitive breaks; users keep every other extra.
- New 'all minus known-broken' tier between [all] and the existing
  PyPI-only-extras tier. Only kicks in when [all] fails resolve.
- All three tiers explicit: every fallback announces which tier
  landed and prints a re-run hint when not on Tier 1.
- install.ps1 and install.sh both regenerate their tier specs from
  the same _BROKEN_EXTRAS array so updates stay in sync.

Side effect: install.ps1 Tier 2 spec previously hardcoded 'mistral'
in its extra list — bug fixed by the refactor (mistral is filtered
out).

# Config

hermes_cli/config.py — DEFAULT_CONFIG.security gains:
- acked_advisories: []  (advisory IDs the user has dismissed)
- allow_lazy_installs: True  (security gate for ensure())

No config version bump needed — both keys nest under existing
security: block, and load_config's deep-merge picks up DEFAULT_CONFIG
defaults for users with older configs.

# Tests

tests/hermes_cli/test_security_advisories.py — 23 tests covering:
- detect_compromised matches/non-matches, wildcard frozenset
- ack persistence, idempotence, blank rejection, config-failure path
- banner cache rate limiting + 24h re-banner + ack-stops-banner
- short_banner_lines / full_remediation_text / render_doctor_section /
  gateway_log_message
- shipped catalog well-formedness invariant

tests/tools/test_lazy_deps.py — 40 tests covering:
- spec safety: 11 safe parametrized + 18 unsafe parametrized
- allowlist: unknown-feature rejection, namespace.name shape,
  every shipped spec passes the safety regex
- security gating: config flag, env var, default, fail-open
- ensure() happy/sad paths: already-satisfied, install success,
  pip stderr surfaced on failure, install-succeeds-but-still-missing
- is_available, feature_install_command

Combined: 63 new tests, all passing under scripts/run_tests.sh.

# Validation

- scripts/run_tests.sh tests/hermes_cli/test_security_advisories.py
  tests/tools/test_lazy_deps.py → 63/63 passing
- scripts/run_tests.sh tests/hermes_cli/test_doctor.py
  tests/hermes_cli/test_doctor_command_install.py
  tests/tools/test_tts_mistral.py tests/tools/test_transcription_tools.py
  tests/tools/test_transcription_dotenv_fallback.py → 165/165 passing
- scripts/run_tests.sh tests/hermes_cli/ tests/tools/ →
  9191 passed, 8 pre-existing failures (verified on origin/main
  before this change)
- bash -n on install.sh and setup-hermes.sh → OK
- py_compile on all modified .py files → OK
- End-to-end smoke test of detect_compromised + render_doctor_section
  + gateway_log_message with mocked installed version → produces
  copy-pasteable remediation output

# Community

Full advisory + remediation steps:
website/docs/community/security-advisories/shai-hulud-mistralai-2026-05.md

Short-form post drafts (Discord, GitHub pinned issue, README banner):
scripts/community-announcement-shai-hulud.md

Refs: PR #24205 (mistral disabled), Socket Security advisory
<https://socket.dev/blog/mini-shai-hulud-worm-pypi>

* build(deps): pin every direct dep to ==X.Y.Z (no ranges)

Companion to the supply-chain advisory work: replace every >=/</~= range
in pyproject.toml's [project.dependencies] and [project.optional-dependencies]
with an exact ==X.Y.Z pin sourced from uv.lock.

Why: ranges allow PyPI to ship a fresh version of any direct dep at any
time without a code review on our side. With ranges, the malicious
mistralai 2.4.6 release would have been pulled by every fresh
'pip install -e .[all]' for the hours between upload and PyPI's
quarantine — exactly the install window we got hit on. Exact pins close
that window: the only way a new package version reaches a user is via
an intentional update on our end.

What the user-facing change is: nothing, behavior-wise. Every package
resolves to the same version it was already resolving to via uv.lock —
the pins just remove the resolver's freedom to pick a different one.

Cost: any user installing Hermes alongside another package that requires
a newer pin gets a resolver conflict. Acceptable for our isolated-venv
install path; documented in the new comment block.

Build-system requires line (setuptools>=61.0) is intentionally left
as a range — pinning the build backend would block fresh pip from
bootstrapping the build on architectures where that exact wheel isn't
available.

mistral extra (mistralai==2.3.0) is pinned but stays out of [all]
(per PR #24205). 'uv lock' regeneration will fail until PyPI restores
mistralai; lockfile regeneration is gated behind that, NOT on every PR.

LAZY_DEPS in tools/lazy_deps.py also moved to exact pins so the lazy-
install pathway can never resolve a different version than the one
declared in pyproject.toml.

Validation:

- Cross-checked all 77 pinned direct deps in pyproject.toml against
  uv.lock — every pin matches the resolved version exactly.
- Cross-checked all LAZY_DEPS specs against uv.lock — same.
- 'uv pip install -e .[all] --dry-run' resolves 205 packages cleanly.
- tests/tools/test_lazy_deps.py + tests/hermes_cli/test_security_advisories.py
  → 63/63 passing (every shipped spec passes the safety regex).
- Doctor + TTS + transcription targeted suite → 146/146 passing.

* build(deps): hash-verify transitives via uv.lock; remove unresolvable [mistral] extra

You asked: 'what about the dependencies the dependencies rely on?' —
correctly noting that exact-pinning direct deps in pyproject.toml does
NOT cover the transitive graph. `pip install` and `uv pip install` both
re-resolve transitives fresh from PyPI at install time, so a compromised
transitive (e.g. `httpcore` if it got worm-poisoned tomorrow) would
still hit our users even with every direct dep exact-pinned.

# What this commit fixes

1. **Both real installer scripts now prefer `uv sync --locked` as Tier 0.**
   uv.lock records SHA256 hashes for every transitive — a compromised
   package with a different hash gets REJECTED. Falls through to the
   existing `uv pip install` cascade if the lockfile is missing or
   stale, with a loud warning that the fallback path does NOT
   hash-verify transitives. Previously only `setup-hermes.sh` (the dev
   path) used the lockfile; `scripts/install.sh` and `scripts/install.ps1`
   (the paths fresh users actually run) skipped it.

2. **Removed the `[mistral]` extra entirely.** The `mistralai` PyPI
   project is fully quarantined right now — every version returns 404,
   so any pin we wrote was unresolvable, which broke `uv lock --check`
   in CI. Restoration is documented in pyproject.toml as a 5-step
   checklist (verify, re-add extra, re-enable in 4 modules, regenerate
   lock, optionally re-add to [all]).

3. **Regenerated uv.lock.** 262 packages, mistralai/eval-type-backport/
   jsonpath-python pruned. `uv lock --check` now passes.

# Defense-in-depth view

| Layer                      | Where             | Protects against                          |
|----------------------------|-------------------|-------------------------------------------|
| Exact pins in pyproject    | direct deps       | new mistralai 2.4.6-style direct compromise |
| uv.lock + `--locked` install | transitive graph  | transitive worm injection                  |
| Tier-0 hash-verified path  | install.sh / .ps1 | actually USE the lockfile in fresh installs |
| `uv lock --check` CI gate  | every PR          | drift between pyproject and lockfile      |
| `hermes_cli/security_advisories.py` | runtime  | cleanup for users who already got hit      |

The exact pinning + hash verification together close the supply-chain
gap. Without the lockfile path, exact pins alone are theater.

# Validation

- `uv lock --check` → passes (262 packages resolved, no drift).
- `bash -n` on install.sh + setup-hermes.sh → OK.
- 209/209 tests passing across new + adjacent test files
  (test_lazy_deps.py, test_security_advisories.py, test_doctor.py,
  test_tts_mistral.py, test_transcription_tools.py).
- TOML parse OK.

* chore: remove community announcement drafts (PR body covers it)

* build(deps): lazy-install every opt-in backend (anthropic, search, terminal, platforms, dashboard)

Extends the lazy-install framework to cover everything that's not used by
every hermes session. Base install drops from ~60 packages to 45.

Moved out of core dependencies = []:
- anthropic   (only when provider=anthropic native, not via aggregators)
- exa-py, firecrawl-py, parallel-web (search backends; only when picked)
- fal-client  (image gen; only when picked)
- edge-tts    (default TTS but still optional)

New extras in pyproject.toml: [anthropic] [exa] [firecrawl] [parallel-web]
[fal] [edge-tts]. All added to [all].

New LAZY_DEPS entries: provider.anthropic, search.{exa,firecrawl,parallel},
tts.edge, image.fal, memory.hindsight, platform.{telegram,discord,matrix},
terminal.{modal,daytona,vercel}, tool.dashboard.

Each import site now calls ensure() before importing the SDK. Where the
module had a top-level try/except (telegram, discord, fastapi), the
graceful-fallback pattern was extended to lazy-install on first
check_*_requirements() call and re-bind module globals.

Updated test_windows_native_support.py tzdata check from snapshot
(>=2023.3 literal) to invariant (any version + win32 marker).

Validation:
- Base install: 45 packages (was ~60); 6 newly-extracted packages absent
- uv lock --check: passes (262 packages, no drift)
- 209/209 lazy_deps + advisory + doctor + tts/transcription tests passing
- py_compile clean on all 12 modified modules

website/docs/community/security-advisories/shai-hulud-mistralai-2026-05.md

@@ -0,0 +1,138 @@
+# Hermes Agent — Security Advisory: Mini Shai-Hulud worm (mistralai 2.4.6)
+
+**Date:** May 12, 2026
+**Status:** Quarantined upstream / mitigated in Hermes
+**Severity:** Critical
+**Affected:** Users who installed `hermes-agent[all]` or `hermes-agent[mistral]` between the upload of `mistralai 2.4.6` and PyPI's quarantine of the package.
+
+## What happened
+
+The Mini Shai-Hulud supply-chain worm crossed from npm to PyPI on 2026-05-12.
+Among the compromised PyPI artifacts was `mistralai 2.4.6` — the official
+Mistral AI Python SDK. The worm steals credentials from environment
+variables and credential files (`~/.npmrc`, `~/.pypirc`, `~/.aws/credentials`,
+GitHub PATs, cloud SDK tokens) and exfils them to a hardcoded webhook.
+
+Hermes Agent listed `mistralai>=2.3.0,<3` as the runtime dependency for its
+optional Mistral TTS / STT providers. Users who installed
+`pip install -e ".[all]"` between the malicious upload and the quarantine
+pulled `mistralai 2.4.6` into their venv. PyPI has since removed the project
+(`pypi:project-status: quarantined`), so the package is no longer
+installable, but copies that landed before quarantine remain in users'
+environments.
+
+## Am I affected?
+
+Run on the host where you installed Hermes:
+
+```bash
+hermes doctor
+```
+
+If the **Security Advisories** section flags
+`mistralai==2.4.6`, you have the compromised package and must remediate.
+If it flags any **other** version of `mistralai`, you are not on the
+compromised release — but we still recommend uninstalling, since the
+project is currently quarantined and we have disabled Mistral TTS / STT
+in Hermes regardless.
+
+You can also check manually:
+
+```bash
+pip show mistralai 2>/dev/null | grep -i version
+```
+
+## What we've done in Hermes Agent
+
+1. **Removed `mistral` from the `[all]` extra** so fresh installs no
+   longer pull the package by default. (PR #24205, already on main.)
+2. **Disabled the Mistral TTS and STT providers** in the runtime — they
+   return a "temporarily disabled" error and won't import the SDK even
+   if the venv still has it.
+3. **Added a security advisory checker** (`hermes doctor` and CLI startup
+   banner) that detects `mistralai 2.4.6` if it's still installed and
+   surfaces remediation steps. The banner is rate-limited (max once per
+   24h per advisory) and dismissible via `hermes doctor --ack`.
+4. **Hardened the installer fallback tiers.** When one extra's
+   dependency becomes unavailable on PyPI, the installer now degrades
+   gracefully — keeping every other extra — instead of dropping all the
+   way to a stripped install. Future supply-chain incidents won't
+   silently demote users.
+5. **Added a lazy-install framework** (`tools/lazy_deps.py`) so opt-in
+   backends (Mistral, ElevenLabs, Honcho, etc.) can be installed on
+   demand when the user enables them, rather than eagerly at install
+   time. This shrinks every fresh install's blast radius for future
+   single-package compromises.
+
+## What you should do
+
+If `hermes doctor` flags `mistralai==2.4.6`, treat the credentials in
+your environment as exposed:
+
+1. **Uninstall the compromised package:**
+   ```bash
+   pip uninstall -y mistralai
+   # or, if you installed via uv:
+   uv pip uninstall mistralai
+   ```
+
+2. **Rotate API keys.** Every key in `~/.hermes/.env` should be rotated:
+   OpenRouter, Anthropic, OpenAI, Nous, GitHub, AWS, Google, Mistral,
+   and any other provider tokens you have configured. If you used a
+   shell that exported keys (`.bashrc`, `.zshrc`, etc.), rotate those
+   too.
+
+3. **Audit credential files** for tokens that may have been read:
+   `~/.npmrc`, `~/.pypirc`, `~/.aws/credentials`, `~/.config/gh/hosts.yml`,
+   `~/.docker/config.json`, `~/.kube/config`, `~/.ssh/`. The worm
+   harvested files matching these patterns.
+
+4. **Check GitHub** for unexpected new SSH keys, deploy keys, or webhook
+   additions on repositories you have admin on. The worm uses stolen
+   GitHub tokens to add backdoors.
+
+5. **After cleanup**, dismiss the Hermes warning:
+   ```bash
+   hermes doctor --ack shai-hulud-2026-05
+   ```
+
+## When will Mistral TTS / STT come back?
+
+When PyPI restores the `mistralai` project to a clean release and we
+verify the new release on a clean network, we will re-enable Mistral
+TTS / STT in Hermes Agent. Until then, use Edge TTS (default, no key),
+ElevenLabs, OpenAI TTS, MiniMax TTS, or any of the user-defined command
+providers. For STT, use Groq Whisper or OpenAI Whisper.
+
+## Future hardening
+
+This incident exposed two structural weaknesses in our install path:
+
+- Eager-install of every optional extra meant ONE compromised package
+  could break the whole `[all]` resolve. **Fixed** via tiered fallback +
+  lazy-install framework.
+- Users had no way to know whether they had a poisoned dependency.
+  **Fixed** via `hermes_cli/security_advisories.py` and the
+  `hermes doctor` integration.
+
+We will continue to extend `tools/lazy_deps.py` so additional opt-in
+backends (Slack, Matrix, Bedrock, DingTalk, Feishu, Google Workspace,
+YouTube transcripts, etc.) can be installed on first use rather than
+eagerly. This reduces the blast radius of any future single-package
+compromise.
+
+## References
+
+- Socket Security report: <https://socket.dev/blog/mini-shai-hulud-worm-pypi>
+- PyPI quarantine: <https://pypi.org/simple/mistralai/>
+  (project-status: quarantined as of 2026-05-12)
+- Hermes Agent PR (mistral disabled): #24205
+- Hermes Agent PR (advisory checker + lazy installs): _this PR_
+- GitHub security advisory: _to be filed alongside this PR_
+
+## Credits
+
+Reported via [@SocketSecurity](https://twitter.com/SocketSecurity) and
+the broader supply-chain security community. Hermes Agent's response
+(detection, lazy-install framework, installer tier hardening) was built
+by the Hermes Agent team at Nous Research.