Reworks the salvaged command module into a CommandSource(SecretSource)
registered as the third bundled source, composing with Bitwarden and
1Password through the apply_all() orchestrator — enable any combination
simultaneously. The original PR's secrets.provider single-selector is
deliberately dropped: multi-source is first-class and a mutually
exclusive provider switch would regress that.
- fetch() only fetches; precedence/override/conflicts/environ writes stay
in the orchestrator. ErrorKind classification + remediation hints.
- apply_command_secrets() kept as a legacy shim (parser/security helpers
unchanged: HERMES_SECRET_KEY data-only key passing, cross-key misroute
guard, base64-padding disambiguation, timeout + output cap, structured-
fields-only failure logging, stderr discarded).
- Dispatch tests rewritten for the registry path incl. an explicit
two-sources-compose test; selector tests removed with the selector.
- cli-config.yaml.example + docs page (command.md), secrets index entry.
- contributors mapping for mvalentin@valensys.net -> 0xr00tf3rr3t.
Brings the agent's secret-source system to parity with the desktop app's
`command` secrets provider (hermes-desktop src/main/secrets/commandProvider.ts),
so a vault helper configured for the desktop also resolves on the gateway/CLI.
NEW agent/secret_sources/command.py — ports the TS provider's security model:
- Runs a user-configured helper via `/bin/sh -c`; the requested key travels
ONLY in the HERMES_SECRET_KEY env var, never interpolated into the command
string, so a hostile key name is inert data (not code).
- parse_secret_output mirrors the TS parser: exact dotenv-key match wins; >=2
env-shaped lines without the wanted key -> None; otherwise a bare value;
base64 '='-padding disambiguation; cross-key misroute guard (a single
OTHER_KEY=realvalue line never leaks into a different wanted key).
- Hard 3s timeout (kills the whole process group via killpg, so a forking
helper can't keep the pipe open), 1 MiB output cap, POSIX-only (Windows
degrades to an empty result + warning). Every failure degrades to "no value";
it never raises and never blocks startup.
- Logs ONLY structured fields (code=/signal=/errno=) to stderr; the helper's
stderr is piped and DISCARDED; the command string and secret values are
never logged. Reuses bitwarden.py's FetchResult so env_loader consumes both
sources identically.
hermes_cli/env_loader.py — _apply_external_secret_sources now reads a unified
`secrets.provider` selector ("env" | "command" | "bitwarden"):
- provider=command routes to apply_command_secrets, records the provenance as
"command" in _SECRET_SOURCES (so format_secret_source_suffix labels keys
"(from command)" — already generic, not duplicated), and re-runs the ASCII
credential sanitizer like the bitwarden path.
- provider=bitwarden keeps the existing behavior byte-for-byte.
- env / unset is a no-op (today's default — zero change for existing users).
- BACK-COMPAT: a config with only `secrets.bitwarden.enabled: true` and no
`provider` key is treated as provider=bitwarden, so existing Bitwarden users
are unaffected.
Config (the provider selector, command path, timeouts) lives in config.yaml
under `secrets:` per the project rubric — only resolved secret VALUES touch env.
Tests: NEW tests/test_command_secret_source.py — 27 cases, E2E against a real
temp HERMES_HOME with real chmod+x shell helpers (not mocks): bare/dotenv/
base64 round-trip, cross-key misroute, injection-inert key (canary not
created), timeout kill within bound, non-zero-exit degrade, no-secret-in-logs,
precedence/override, dispatch via config.yaml provider:command, idempotency,
and back-compat bitwarden routing. 27 new + 50 baseline green; wider
secrets/env_loader/config surface 229 passed / 5 skipped, no regression.
Fixes the profile-clobber bug cluster at the apply_all() chokepoint so
every secret source — bundled and plugin — gets both behaviors for free:
- secrets.preserve_existing (#58073): env var names whose existing .env /
shell value always wins, even against a source with
override_existing: true. Escape hatch for per-profile platform
secrets while everything else rotates centrally.
- Profile aliasing (#51447): under a named profile, an applied
FOO_<PROFILE> var (credential-shaped suffixes only) also hydrates the
canonical FOO, so adapters/plugins that read fixed env names see the
profile's value. Direct supply beats alias; protected/claimed/
override guards all apply; secrets.profile_alias: false disables.
Reimplements the intent of PR #58085 (tianma-if, preserve_existing on the
legacy Bitwarden apply shim) and PR #51616 (LeonSGP43, profile aliasing
inside the Bitwarden backend) on the SecretSource orchestrator that
superseded those code paths.
Fixes#58073. Fixes#51447.
Co-authored-by: tianma-if <5895871+tianma-if@users.noreply.github.com>
Co-authored-by: LeonSGP43 <154585401+LeonSGP43@users.noreply.github.com>
The 1Password secret source builds a minimal allowlisted environment for the
`op read` child process. The allowlist omits OP_LOAD_DESKTOP_APP_SETTINGS, so a
user who exports it (shell, .env, or service unit) sees it silently stripped
before it reaches `op`.
That var is `op`'s documented switch to skip the desktop-app integration probe.
When the 1Password desktop app is installed, `op` probes its settings/socket at
startup *before* evaluating service-account auth. If the desktop app's group
container is wedged (e.g. macOS 'Interrupted system call' on the 1Password group
container), that probe blocks with no timeout, so `op read` hangs indefinitely
even with a valid OP_SERVICE_ACCOUNT_TOKEN present. Setting
OP_LOAD_DESKTOP_APP_SETTINGS=false is the intended escape hatch — but stripping
it means it has no effect on exactly the headless boxes that need it.
Fix: add OP_LOAD_DESKTOP_APP_SETTINGS to _OP_ENV_ALLOWLIST so the documented
var reaches the child. No behavior change when it's unset. Adds a focused test
alongside the existing allowlist test.
Repro: on a machine with a wedged 1Password desktop container + a valid SA
token, `op read` hangs 600s+ without the var and returns in ~4s with it — but
only if it actually reaches the op process, which this allowlist entry ensures.
Co-authored-by: Minh Nguyen <menhguin@users.noreply.github.com>
_auth_fingerprint() built the 1Password secret cache-key from the
service-account token, OP_ACCOUNT, and OP_SESSION_* vars but omitted
OP_CONNECT_HOST/OP_CONNECT_TOKEN, which are in _OP_ENV_ALLOWLIST and are
forwarded to the op child (the Connect-server auth path). Rotating
OP_CONNECT_TOKEN or re-pointing OP_CONNECT_HOST at a different Connect
identity left the fingerprint unchanged, so both the in-process and disk
caches kept serving secrets resolved under the old Connect credentials for
the full TTL (default 300s, disk-persisted across invocations). This
contradicts the function's own docstring invariant that a value cached
under a previous identity is never served under a new one; it closes the
gap for the Connect path, matching the OP_SESSION_*/service-account paths
that are already protected.
The stale-fallback branch called _read_disk_cache(), a helper removed in
db495b0fba when disk-cache logic moved to the
shared DiskCache class — every fallback attempt raised NameError instead of
serving cached secrets, silently defeating the PR's whole purpose. Port to
_DISK_CACHE.read().
Also tighten the fallback per DiskCache's TTL contract and the secret-source
error taxonomy:
- Gate on cache_ttl_seconds > 0 so a caller that opted out of caching
entirely (ttl=0) never gets a secret value that didn't come from a live
fetch, even on the failure path.
- Gate on _classify_bws_error(str(exc)) being NETWORK or TIMEOUT, reusing
the existing classifier — an AUTH_FAILED or malformed-output failure must
still raise, since serving stale secrets there would mask a real
credential/config problem instead of a transient outage.
Ported the test helpers off the removed _write_disk_cache to a direct JSON
write (matching this file's existing disk-cache test convention) and added
tests for the auth-failure, malformed-output, and zero-TTL gates. Reverting
the fix and re-running confirms 7 of 8 stale-fallback tests fail with the
original NameError.
Without this, a single DNS hiccup or BWS outage at gateway startup leaves
the whole fleet running with an empty credential pool — every model call
fails until someone restarts after the network recovers. When a previous
successful fetch already populated the disk cache, return those secrets
with an explicit warning instead of raising RuntimeError.
`use_cache=False` (explicit opt-out) still raises so manual flows like
the setup wizard surface the original error. The disk cache is not
re-written on the fallback path so a process restart still triggers a
proper TTL re-check.
Fixes#41925
* feat(secrets): one-command token rotation + actionable startup errors for all secret sources
When a Bitwarden machine-account token expired, users saw a raw Rust
error dump (invalid_client + Location: + backtrace hints) and the only
fix was manually editing .env or re-running the whole setup wizard.
- New `hermes secrets bitwarden token` / `hermes secrets onepassword
token`: paste a new token (masked prompt or flag), the command probes
the backend BEFORE persisting — a rejected token changes nothing; a
good one is written to .env and the fetch caches are cleared.
- New optional SecretSource.remediation(kind, cfg) hook: startup
warnings now print a '→ Run `hermes secrets <name> token`…' fix-it
line after any fetch error, for bundled AND plugin sources (generic
per-ErrorKind defaults in the ABC).
- bws stderr is summarized to its cause line (Location:/backtrace noise
dropped) and invalid_client/invalid_grant/400 identity rejects are
now classified AUTH_FAILED (was INTERNAL) with a plain-English
explanation naming the token env var.
- op whoami probe accepts a candidate token so rotation validates the
NEW credential, not the ambient one.
Additive hook with defaults — no SECRET_SOURCE_API_VERSION bump.
* docs: fix MDX parse error in secret-source-plugin hook table
Escaped backticks around a <name> placeholder made MDX parse it as an
unclosed JSX tag, breaking the docs-site build. Use a plain code span
instead.
Follow-up on the cherry-picked #36896 commits, wiring 1Password into
the new registry as the reference *mapped* source:
- OnePasswordSource adapter (shape=mapped, scheme=op): fetch-only —
precedence, override semantics, conflict warnings, and env writes
move to the orchestrator; apply_onepassword_secrets kept as legacy
shim like Bitwarden's.
- Registered in _ensure_builtin_sources; mapped op:// bindings now
outrank bulk Bitwarden project dumps on contested vars.
- _cache.py FetchResult/is_valid_env_name re-exported from base so
there is exactly one canonical definition; bitwarden.py re-adapted
onto the contributor's DiskCache substrate.
- ErrorKind classification for op failures (auth/binary/empty/network).
- Registry + conformance coverage for OnePasswordSource, incl. the
headline multi-source test: both vaults claim the same var, mapped
1Password wins, conflict surfaced, provenance correct.
- env_loader tests migrated off the legacy apply_* mocks onto the
fetch layer; AUTHOR_MAP entry for @hwrdprkns.
Resolve provider credentials from 1Password op://vault/item/field references
at startup via the official `op` CLI, alongside the existing Bitwarden source.
Users map env-var names to references in secrets.onepassword.env; after .env
loads, each is resolved with `op read` and injected into os.environ. Auth is
whatever `op` already uses (service-account token or desktop/interactive
session) — Hermes never authenticates or installs `op` itself.
Startup-safe and fail-open: a missing binary, expired auth, a bad reference,
or an empty value each warn and fall back to existing credentials, never
blocking startup. Successful, complete pulls are cached in-process and on disk
(<hermes_home>/cache/op_cache.json, 0600) via the shared DiskCache; only
secret values are stored, never the token (auth is fingerprinted into the
key). Adds `hermes secrets onepassword {setup,status,set,remove,sync,disable}`
(aliases op/1password), config defaults, the cli-config example, docs, and
hermetic tests.
Hardening applied across both backends in env_loader: each source runs in its
own guard, config sections are coerced to dict, and cache_ttl_seconds is
coerced defensively — so a malformed secrets: section can't abort startup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull the disk-cache + FetchResult substrate out of bitwarden.py into a new
agent/secret_sources/_cache.py: FetchResult, CachedFetch, is_valid_env_name,
and a generic DiskCache (atomic mkstemp -> chmod 0600 -> os.replace write,
0700 cache dir, TTL-gated read AND write). Bitwarden now consumes it via a
module-level DiskCache instance and thin wrappers, so the security-sensitive
atomic-write/0600/TTL logic lives in exactly one place instead of being
copy-pasted per backend (and drifting). Behavior is unchanged — the full
Bitwarden suite passes untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When Hermes runs in TUI mode, the gateway child process communicates with
the Node.js parent over a JSON-RPC protocol on stdin. Subprocess calls that
inherit this stdin fd can trigger a race condition where the child's stdin
read returns EOF, causing the gateway to exit cleanly (exit code 0) mid-tool-
execution.
This is the same root cause as issue #14036 (byterover plugin) and PR #39257
(SSH environment backend). This commit applies the fix — stdin=subprocess.DEVNULL
— to all 85 subprocess.run() and subprocess.Popen() calls that execute inside
the TUI gateway child process.
Scope: TUI-context code only (agent/, tools/, plugins/, tui_gateway/server.py).
CLI code (cli.py, hermes_cli/), tests, scripts, and gateway process management
are excluded — they don't run inside the TUI child and inherit the terminal's
stdin, not the JSON-RPC pipe.
85 call sites across 28 files. All files pass syntax check.
Remove unused imports (F401) and duplicate/shadowed import
redefinitions (F811) across the codebase using ruff's safe
autofixes. No behavioral changes -- imports only.
- ~1400 safe autofixes applied across 644 files (net -1072 lines)
- __init__.py re-exports preserved (excluded from F401 removal so
public re-export surfaces stay intact)
- Re-exports that are imported or monkeypatched by tests but look
unused in their defining module are kept with explicit # noqa:
F401 (gateway/run.py load_dotenv; run_agent re-exports from
agent.message_sanitization, agent.context_compressor,
agent.retry_utils, agent.prompt_builder, agent.process_bootstrap,
agent.codex_responses_adapter)
- Unsafe F841 (unused-variable) fixes deliberately skipped -- those
can change behavior when the RHS has side effects
- ruff lints remain disabled in pyproject.toml (only PLW1514 is
selected); this is a one-time cleanup, not a config change
Verification:
- python -m compileall: clean
- pytest --collect-only: all 27161 tests collect (zero import errors)
- core entry points import clean (run_agent, model_tools, cli,
toolsets, hermes_state, batch_runner, gateway)
- static scan: every name any test imports directly from an edited
module still resolves
* perf(bitwarden): persist secret-fetch cache across CLI invocations
Every `hermes` invocation paid a ~380ms tax for `bws secret list` to
Bitwarden Secrets Manager because the existing cache was in-process only.
Back-to-back `hermes chat -q`, gateway-spawned agents, and cron-launched
runs all re-fetched.
Adds a disk-persisted L2 cache at `<hermes_home>/cache/bws_cache.json`
(mode 0600, never contains the access token — only the SHA-256
fingerprint prefix). Same TTL as the in-process cache. Read on miss,
write on bws success, ignored on key mismatch / corruption / expiry.
Measured on a startup profile:
load_hermes_dotenv() cold: 372ms → warm (disk cache hit): 20ms
End-to-end `hermes --version` cold→warm: 666ms → ~295ms.
In a hermes-vs-codex benchmark across 11 single- and multi-turn tasks
(framework overhead = wall − llm − tool_exec, median over 3 trials):
cohort before after saved
single-turn (median) 2.96s 2.31s -0.65s
multi-turn (5-turn) 9.40s 8.95s -0.45s (≈0.3s/turn)
Hermes now wins head-to-head on 6/11 tasks vs codex (was 4/11 before).
The remaining ~0.6s single-turn delta is mostly Python's own import
cost in hermes_cli.main, which is a separate optimization.
* perf(cli): lazy-load model catalog + dedupe config.yaml reads at startup
Two import-time wins on top of the bws disk-cache fix:
1. Lazy-load `hermes_cli.models._PROVIDER_MODELS` via PEP 562
module-level `__getattr__`. The catalog is ~55ms of work that was
eagerly imported on every CLI invocation (line 4557 `if not
_is_termux_startup_environment(): from hermes_cli.models import
_PROVIDER_MODELS`). Audit showed every internal call site already
does its own function-local import; only test code reads
`hermes_cli.main._PROVIDER_MODELS` as a module attribute, and
__getattr__ keeps that working transparently. First access triggers
the import once and caches the result on the module via
`globals()[name] = ...`, so subsequent reads are dict lookups.
2. Dedupe the double config.yaml read in the top-of-module bootstrap.
Previously: one raw yaml.safe_load for the `security.redact_secrets`
bridge, then a separate full `load_config()` (with deep-merge) for
`network.force_ipv4`. Both keys come from the same file. Merged
into one raw yaml load.
Combined with the bws cache fix in the previous commit:
hermes --version wall time:
original (cold): 666 ms
after bws fix (warm): 295 ms
after lazy-load + dedupe: 228 ms (-67 ms additional, -66% from original)
Tests:
- tests/hermes_cli/test_api_key_providers.py: 173/173 pass
(lazy __getattr__ correctly handles
`from hermes_cli.main import _PROVIDER_MODELS`)
- tests/test_ipv4_preference.py + tests/hermes_cli/test_redact_config_bridge.py +
tests/agent/test_redact.py: 93/93 pass (dedupe preserves both bridges)
- tests/test_bitwarden_secrets.py + env_loader tests: 49/49 pass
Closes#31370.
bws defaults to the US identity endpoint, so EU Cloud and self-hosted
machine-account tokens fail with [400 Bad Request] {"error":"invalid_client"}
during 'hermes secrets bitwarden setup'. The token is valid — it's just
being checked against the wrong region.
Add a Bitwarden region step to the wizard between the access-token and
project-list steps:
Step 1 Install bws
Step 2 Provide access token
Step 3 Pick region <-- new (US / EU / self-hosted-custom-URL)
Step 4 Pick project (now talks to the right endpoint)
Step 5 Test fetch
Region is stored in config.yaml as secrets.bitwarden.server_url and
plumbed into every bws subprocess as BWS_SERVER_URL (project list,
secret list, test fetch, and the env_loader startup pull).
Also:
- Non-interactive: 'hermes secrets bitwarden setup --server-url ...'
- Pre-existing BWS_SERVER_URL in the shell is detected and reused
- Cache key includes server_url so EU/US fetches don't collide
- 'hermes secrets bitwarden status' shows the configured region
- 'invalid_client' / '400 Bad Request' from bws now triggers a hint
pointing at the region setting instead of looking like a bad token
* feat(secrets): Bitwarden Secrets Manager integration with lazy bws install
Pull API keys from Bitwarden Secrets Manager at process startup
instead of storing them all in plaintext in ~/.hermes/.env. One
bootstrap token (BWS_ACCESS_TOKEN) replaces N per-provider keys, and
rotating a credential becomes a single change in the Bitwarden web
app.
Bitwarden defaults to source of truth: secrets pulled from BSM
overwrite any matching env vars on startup so rotations actually
take effect. Set secrets.bitwarden.override_existing: false in
config.yaml to invert.
The bws binary is auto-downloaded into ~/.hermes/bin/bws on first
use (pinned to v2.0.0, SHA-256 verified against the GitHub release
checksum file). No apt, brew, or sudo required.
New surfaces:
hermes secrets bitwarden setup — interactive wizard
hermes secrets bitwarden status — config + binary + token state
hermes secrets bitwarden sync — dry-run fetch / --apply exports
hermes secrets bitwarden disable — flip enabled: false
hermes secrets bitwarden install — just download the binary
Failures (missing binary, bad token, no network) never block Hermes
startup — they emit a one-line warning to stderr and continue with
whatever credentials .env already had.
Docs: website/docs/user-guide/secrets/{index,bitwarden}.md
Tests: tests/test_bitwarden_secrets.py (26 tests, hermetic — bws
subprocess and HTTP downloads fully mocked)
* chore(infographic): add bitwarden-secrets-manager bento-grid retro-pop-grid
Generated for PR #30035 — Bitwarden Secrets Manager integration.
Style picked via pick_pr_infographic_style.py rotation:
layout: bento-grid
style: retro-pop-grid
aspect: 1:1 square
Saved at infographic/bitwarden-secrets-manager/infographic.png
