Reads header bytes 28-31 after every COMMIT and compares against actual file size. Raises sqlite3.DatabaseError on torn-extend (actual_pages < page_count). Also sets PRAGMA wal_autocheckpoint=100 in connect().
Refs: #31208 (Bug E - same file, coordinate), #30973 (wal_autocheckpoint)
Refs: #30445, #30896, #30908 (corruption reports)
`detect_crashed_workers` calls `_pid_alive` on every `running` task whose
claim is held by this host. The check can transiently return False for a
freshly-spawned worker (fork → /proc-visibility lag, or reap-race
between SIGCHLD and parent reaping). When a second dispatcher ticks
inside that window it reclaims the task and spawns a duplicate worker.
Add `DEFAULT_CRASH_GRACE_SECONDS = 30` and an
`HERMES_KANBAN_CRASH_GRACE_SECONDS` env-var override.
`detect_crashed_workers` skips the liveness check when
`time.time() - started_at < grace`. The existing 15-minute claim TTL
still reclaims genuinely-crashed workers; grace only suppresses the
launch-window false positive.
`HERMES_KANBAN_CRASH_GRACE_SECONDS=0` is set on the `kanban_home`
fixture in `test_kanban_core_functionality.py` so existing tests that
assert immediate reclaim retain pre-fix semantics.
Companion to merged PR #23442 (`release_stale_claims`, closes#23025),
which addressed the same multi-dispatcher race in the stale-claim path.
Related: #20015 (`_pid_alive` false-negative behaviour),
When code inside a write_txn block raises an OperationalError that SQLite
has already auto-rolled-back (typical for disk I/O error,
database is locked, and database disk image is malformed), the
explicit ROLLBACK in write_txn.__exit__ itself raises
cannot rollback - no transaction is active and the secondary exception
replaces the original in the traceback. Operators see a misleading error
and lose the diagnostic information they need.
Swallow the rollback-time OperationalError so the caller always sees the
original cause.
Confirmed reproducer: tests/hermes_cli/test_kanban_db.py::
test_write_txn_preserves_original_exception_when_rollback_fails
apply_wal_with_fallback() treated "disk i/o error" as a permanent
WAL-incompatibility marker, identical to "locking protocol" (NFS) and
"not authorized" (FUSE). But EIO during PRAGMA journal_mode=WAL is
typically TRANSIENT — page-cache pressure, brief lock contention,
recoverable storage hiccups — not a permanent filesystem property.
Treating transient EIO as a permanent downgrade signal produces the
mixed-journal-mode-across-processes corruption pattern:
1. Process A opens kanban.db, hits transient EIO on the WAL pragma,
silently downgrades to journal_mode=DELETE.
2. Process B (no EIO) opens the same file moments later and
successfully sets journal_mode=WAL.
3. A writes rollback-journal frames while B writes WAL frames. SQLite
documents this as unsupported and corrupts the file:
https://www.sqlite.org/wal.html ("all connections to the same
database must use the same locking protocol").
This was the root cause of repeated kanban.db corruption on hosts with
multiple gateway processes plus CLI invocations against the same DB
(observed pattern: corruption shortly after gateway startup, after the
process logged "WAL journal_mode unsupported on this filesystem (disk
I/O error) — falling back to journal_mode=DELETE"). The fallback
warning told the truth — fallback DID happen — but the premise
("unsupported on this filesystem") was wrong; the EIO was a one-shot
event and sibling processes successfully used WAL.
Fix has two layers:
1. Remove "disk i/o error" from _WAL_INCOMPAT_MARKERS. EIO now re-raises
so callers can retry instead of silently corrupting the DB. The two
remaining markers ("locking protocol", "not authorized") are
deterministic per filesystem so they remain safe permanent-downgrade
signals.
2. Belt-and-suspenders: before downgrading on ANY marker match, peek the
on-disk journal mode. If the header says WAL, refuse to downgrade and
re-raise the original error. This guards against any future addition
to _WAL_INCOMPAT_MARKERS turning out to be transient in some
environment we haven't yet seen.
Tests:
- tests/test_hermes_state_wal_fallback.py:
* Flipped test_falls_back_on_disk_io_error → test_reraises_on_disk_io_error
asserting EIO is re-raised, not silently swallowed.
* Added test_does_not_downgrade_when_disk_says_wal covering the
on-disk-header safety guard for the existing legitimate markers.
- tests/hermes_cli/test_kanban_db.py:
* test_connect_falls_back_to_delete_on_locking_protocol now uses a
truly-fresh DB (instead of the kanban_home fixture which pre-inits
in WAL). On NFS the very first process touching the file legitimately
downgrades; on a file already in WAL the new guard correctly refuses.
A standalone reproducer lives at /tmp/kanban-stress/repro_bugD_eio_wal_downgrade.py
(not committed): without fix the DB silently flips from WAL to DELETE
mid-process; with fix the EIO surfaces and the file stays WAL.
Refs: Bug D in the kanban-corruption investigation series (Bugs A and C
shipped in ebe7374f3 and e02147d5e respectively). Bug D explains every
corruption incident this week including those that survived A's
single-dispatcher mitigation, because every CLI invocation is a
separate process whose WAL pragma can transiently fail.
Production corruption #6 left b-tree pages with zeroed headers but intact old cell content — the Bug E pattern. This fix applies three pragma calls on every connect():
- synchronous=FULL (was NORMAL): closes the WAL-checkpoint reordering window where a crash between WAL commit and main-DB write leaves a partially-written b-tree page header. Cost is <1ms per commit on local SSD; negligible at kanban write volume.
- secure_delete=ON: forces SQLite to zero freed page bytes on disk. If a torn write or hardware fault later corrupts a page, the underlying cell content is zero, so corruption is detectable and no stale rows can resurface as live data.
- cell_size_check=ON: adds a read-side guard so corrupt cells surface as errors at read time rather than as silent wrong-data returns.
All three are connection-scoped and re-applied on every connect(). secure_delete also writes a persistent flag into the DB header on the first call against a fresh DB, making the protection durable across processes for new DBs.
Tests added for all four required cases: each pragma active on a fresh connection, and all three re-applied after close+reopen. Also adds the required negative test (migration path does not reset pragmas).
Self-review follow-ups on the salvage of #22494:
W2 — Added encoding="utf-8" to read_text() calls. scripts/install.sh
contains 48 em-dash ("—") characters and ~1500 non-ASCII bytes total;
on Windows with cp1252 default locale, bare read_text() would raise
UnicodeDecodeError. Project-wide cleanup of the other 11 similar sites
across 5 install_sh test files is deferred to a separate follow-up.
W3 — Bound the branch-containment check by the function body (head
"resolve_install_layout() {" / tail "\n}\n") instead of by "next
`return 0` after the marker". scripts/install.sh has 5 additional
`return 0` statements between resolve_install_layout's first one and
EOF; if a future maintainer hoists the export above another conditional
with its own early-return or inserts an early-return between the marker
and the export, the old assertion still passes while the export is
unreachable. The body-bounded slice makes that class of regression
visible.
Also added more specific assertion messages and a guard for the body
extraction to fail loudly if the function signature ever changes.
When installing as root on Linux with the default FHS layout
(/usr/local/lib/hermes-agent), `uv python install` placed the managed
Python under /root/.local/share/uv/python/, which non-root users cannot
traverse. The shared /usr/local/bin/hermes wrapper then failed for them
with "bad interpreter: Permission denied" when execing the venv python.
Export UV_PYTHON_INSTALL_DIR and UV_PYTHON_BIN_DIR to /usr/local/share/uv/
in the root-FHS branch of resolve_install_layout so the managed Python
is world-readable and the shared wrapper works for any user.
Closes#21457
Asserts that when hermes update runs on a fork whose local HEAD matches
origin/main but commit_count == 0, the early-return path still consults
_sync_with_upstream_if_needed() before printing "Already up to date!".
Locks in the fix from the parent commit so the upstream-sync call cannot
silently regress out of the commit_count == 0 branch.
DEFAULT_CODEX_MODELS shipped three slugs that the chatgpt.com Codex
backend rejects with HTTP 400 'The <slug> model is not supported when
using Codex with a ChatGPT account.' on every account tested live:
gpt-5.2-codex
gpt-5.1-codex-max
gpt-5.1-codex-mini
Live verified against https://chatgpt.com/backend-api/codex/models
which returns gpt-5.5, gpt-5.4, gpt-5.4-mini, gpt-5.3-codex,
gpt-5.3-codex-spark, gpt-5.2 for ChatGPT Pro accounts.
When _fetch_models_from_api fell back to DEFAULT_CODEX_MODELS (offline
first-run, transient API failure) the picker surfaced these dead slugs
and crashed on selection. The forward-compat synthesis table chained
them downstream too.
If OpenAI re-enables them on the OAuth-backed Codex backend, live
discovery will pick them up automatically — the defaults list is only
consulted when live discovery is unavailable.
Test fixture pivoted to use gpt-5.3-codex (templated by 4 entries) as
the synthesis driver so the forward-compat test still exercises the
synthesis path.
Salvages the transport-side fix from #32911 (@xxxigm). Closes#32892.
The openai SDK's responses.stream() / responses.parse() eagerly call
_make_tools(tools), which iterates tools without a None guard. Passing
tools=None raises TypeError: 'NoneType' object is not iterable before
any HTTP request is issued (openai==2.24.0).
PR #33042 already removed responses.stream() from our own Codex call
paths, so the specific iteration crash inside _make_tools is no longer
on the hot path. But the right API contract is to omit tools entirely
when there are no functions to expose — passing tools=None to the
backend is semantically wrong regardless of the SDK's iteration
behavior, and we'd hit it again on any future code path that hasn't
migrated off responses.stream().
This applies the transport-level part of @xxxigm's fix: move
'tools': response_tools into the if response_tools: branch so the
key is omitted when there are no tools, just like tool_choice and
parallel_tool_calls already are. Skips the run_agent.py-side
_strip_sdk_none_iterables helper from their PR — that path is now
obsolete because the SDK helper that needed defending is gone.
Tests
- tests/run_agent/test_codex_no_tools_nonetype.py: 6 tests trimmed
from @xxxigm's original 13-test file. Drops the obsolete tests for
_strip_sdk_none_iterables and _RecordingResponsesStream (helpers
that don't exist on main anymore), keeps the transport behavior
tests + the SDK contract sanity check that ensures we notice if
upstream ever fixes _make_tools(None).
- 6/6 passing locally.
Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>
Salvages the intent of #33136 (@Brixyy) onto current main. The original PR
was written against the pre-refactor monolithic run_agent.py and added a
top-level _is_nonretryable_local_validation_error() helper. Both target
functions have since been extracted to agent/conversation_loop.py:2869,
so the salvage applies the equivalent guard inline at that canonical
location rather than reintroducing the helper.
## Why
After #33042 made our own Codex consumer structurally immune to NoneType
crashes, third-party shims, mocked clients, and any future code path that
hasn't migrated could still surface TypeError: 'NoneType' object is not
iterable as a wire-shape mismatch. The agent loop's classifier currently
treats ALL TypeError as a local programming bug and aborts non-retryable
— users on stale Telegram/gateway turns saw bare "Non-retryable error
(HTTP None)" with no recovery.
This is a provider/SDK shape mismatch, not a local programming bug. The
retry/fallback path should run, not be short-circuited.
## What
agent/conversation_loop.py: extend is_local_validation_error to exclude
TypeErrors whose message matches the NoneType-not-iterable shape (case-
insensitive, both "NoneType" and "not iterable" must appear).
tests/run_agent/test_jsondecodeerror_retryable.py:
- update the mirror predicate to match the production check
- add TestNoneTypeNotIterableIsRetryable class with 3 tests (the basic
shape, message variants, unrelated TypeErrors still abort)
- add TestAgentLoopSourceHasNoneTypeCarveOut to enforce the source-level
invariant matches the test mirror
## Validation
tests/run_agent/test_jsondecodeerror_retryable.py +
tests/run_agent/test_31273_402_not_retried.py → 14/14 passing
Co-authored-by: Brixyy <subrtt@gmail.com>
Closes#33368.
`_CodexCompletionsAdapter.create()` iterates `final.output` from the
Codex Responses stream. The event-driven consumer (introduced in #33042)
always sets `final.output` to a list, so this shape can't come from our
own code path. But:
- Mocked clients in tests can return a typed Response with `output=None`
- Third-party shims / compatibility layers that bypass the consumer can
do the same
- A future code path that wraps a different consumer could regress
The old code `getattr(final, "output", [])` returns `None` (not the
default `[]`) when the attribute EXISTS but is `None`. Iterating
`None` then raises `TypeError: 'NoneType' object is not iterable` —
the exact error logged by title-generation when this fires.
Fix: `getattr(final, "output", None) or []` — single-line defensive
coerce. Cheap; zero risk.
Regression test asserts the auxiliary path handles a final whose
`.output` is `None` (via monkey-patched consumer) without raising and
returns the expected chat.completions-shaped response.
Reporter: @pavegrid-1 (issue #33368).
* feat(image_gen): add Krea provider plugin (Krea 2 Medium + Large)
New built-in image_gen backend wrapping Krea's Krea 2 foundation
image model family. Auto-discovered like the other image_gen plugins
and appears in 'hermes tools' → Image Generation → Krea.
Krea's API is asynchronous — submit returns a job_id, poll /jobs/{id}
until terminal. The provider hides that behind the synchronous
ImageGenProvider.generate() contract: submit, poll every 2s with
light backoff (max 5s), 3-minute ceiling matching Krea's hosted-tool
timeout. Result URL is materialised to $HERMES_HOME/cache/images/
to avoid CDN-expiry 404s downstream (same fix as xAI #26942).
Models:
- krea-2-medium (default — Krea's 'start here' recommendation)
- krea-2-large
Aspect ratios map landscape→16:9, square→1:1, portrait→9:16.
Resolution: 1K (Krea's only current option).
Kwarg passthrough: seed, creativity (raw/low/medium/high), styles,
image_style_references (capped 10), moodboards (capped 1) — matches
Krea's per-request limits. Unknown kwargs are ignored.
Config knobs (config.yaml):
image_gen.provider: krea
image_gen.krea.model: krea-2-medium | krea-2-large
image_gen.krea.creativity: raw | low | medium | high
Env overrides: KREA_API_KEY (required), KREA_IMAGE_MODEL.
KREA_API_KEY is registered in OPTIONAL_ENV_VARS so 'hermes setup'
prompts for it.
31 new tests; image_gen suite + picker + tools_config: 211/211.
* fix(image_gen/krea): address review feedback
- Update KREA_API_KEY setup URL to the canonical token-creation page
(https://www.krea.ai/app/api/tokens). The previous URL returned 404.
- Fail fast on non-retryable HTTP statuses during poll. The previous
loop retried every HTTPError for the full 180s deadline, so an auth
(401), billing (402), forbidden (403), or not-found (404) response
would make image_generate hang for three minutes. Only retry
transient statuses (408/409/425/429/5xx); surface everything else
immediately.
- Add 5 tests covering fail-fast on 401/403/404 and retry on 429/503.
* fix(krea): point users at the real API token dashboard URL
Three call sites linked users to dashboard pages that don't exist:
- hermes_cli/config.py: https://www.krea.ai/app/api/tokens
- plugins/image_gen/krea/__init__.py get_setup_schema: https://www.krea.ai/api-keys
- plugins/image_gen/krea/__init__.py auth_required error: https://www.krea.ai/api-keys
Per Krea's own docs (https://docs.krea.ai/developers/api-keys-and-billing),
the real dashboard URL is https://www.krea.ai/settings/api-tokens. All three
sites now point there.
Use the shared observer/target resolver for session context so peer='user' and explicit configured peer IDs query Honcho from the same assistant-observed perspective when allowed. Add regression coverage for user alias, explicit peer, and self-observer fallback.
honcho_profile(peer="user") returned an empty card even when Honcho
held a populated peer card for the user. Two independent bugs combined
to produce the symptom:
1. Read path: get_peer_card() called _fetch_peer_card(observer, target=user),
which hits GET /peers/{observer}/card?target={user} — the observer's local
card of the user. On self-hosted Honcho v3 this slot is empty unless writes
also use it. The peer card lives on the user peer itself
(GET /peers/{user}/card). Add a fallback: when the observer-target slot is
empty and a target exists, retry against the target peer's own card.
2. Write path: set_peer_card() resolved only the target peer and called
user_peer.set_card(card). The read path uses the assistant peer as
observer, so writes and reads addressed different Honcho card scopes.
Align set_peer_card() with _resolve_observer_target() so writes go to
assistant_peer.set_card(card, target=user_peer_id), matching the read.
Both paths now use the same observer/target resolution, and the read
path additionally falls back to the target's own card for compatibility
with deployments where cards were written directly to the peer.
Closes: related to #13375, #17124, #20729
Three related regressions stemming from the pinUserPeer alias landing:
- Setup wizard read host-only fields when detecting current shape but the
parser supports root-level config and gives host pinUserPeer higher
precedence than pinPeerName. Re-running setup could mis-detect shape
and silently flip routing. Detection now uses the same resolver order
as HonchoClientConfig, and each shape branch scrubs every peer-mapping
key before writing so a stale pinUserPeer=false can't outrank a freshly
written pinPeerName=true. Multi no longer auto-writes
userPeerAliases={} (was silently masking root-level baselines).
- clone_honcho_for_profile inherited pinPeerName but not pinUserPeer, so
a default profile configured with the newer key produced cloned
profiles without the pin.
- Gateway cache-busting signature fingerprinted Honcho user-peer fields
but not ai_peer. Since HonchoSessionManager freezes cfg.ai_peer at
init, mid-flight aiPeer edits kept assistant writes on the old peer
until an unrelated cache eviction. ai_peer is now part of the
signature.
Remove "PR #14984 / #27371 / #1969" references and "the original key /
legacy / backwards-compatible / Port #N" narration from the honcho
plugin README, tests, and one stale code comment. These artefacts age
poorly: they describe how a change happened rather than what the code
does today, and they tax readers who weren't around for the original
work.
Also drop a dangling reference to scratch/memory-plugin-ux-specs.md in
__init__.py — the file isn't in the repo or git history.
No behaviour change.
Three correctness gaps when honcho.json's identity-mapping config changes
mid-flight:
1. The gateway's agent cache signature ignored honcho identity keys, so
editing peerName / pinPeerName / userPeerAliases / runtimePeerPrefix
was silently dropped until an unrelated cache eviction. Extend
_extract_cache_busting_config to fingerprint the resolved honcho
config so the AIAgent rebuilds on the next message.
2. cmd_setup let single → multi flips orphan the pinned-pool history
under peerName without warning. Detect the transition, warn that
runtime users will resolve to fresh empty peers, and auto-steer to
hybrid (alias the operator's runtime IDs back to peerName) so the
operator's own continuity survives. yes / no overrides available.
3. README didn't document the orphaning behaviour. Add a "Migrating
single → multi" callout under Deployment shapes.
Tests:
- TestPinTransition (test_pin_peer_name.py): fresh-manager flip resolves
to runtime, in-process flip is gated by the per-key session cache
(documents the gateway-cache-must-bust contract), 3 cache-bust
signature tests for pin / aliases / prefix.
- TestProfilePeerUniqueness: two profiles pinned to distinct peerNames
resolve to distinct peers; host-level peerName overrides root when
pinned.
- test_single_to_multi_steers_to_hybrid_by_default and
test_single_to_multi_yes_override_keeps_multi (test_cli.py): wizard
guard end-to-end coverage.
The original key 'pinPeerName' from #14984 is ambiguous: a fresh
reader can't tell whether it pins the user peer or the AI peer from
the name alone. The resolver only ever pins the user-side
(_resolve_user_peer_id short-circuits when pin_peer_name is true; the
AI peer is already pinned by construction via aiPeer).
Add 'pinUserPeer' as the canonical alias. Both keys land on the
same internal pin_peer_name field; precedence is host pinUserPeer →
host pinPeerName → root pinUserPeer → root pinPeerName → default.
Host-level always beats root-level regardless of alias, so a host
block can still explicitly disable a root-level pin even via the new
key.
Make _resolve_bool variadic so it can express the four-value
precedence chain. All existing callers pass two positional args +
default keyword, which the new signature accepts unchanged.
Internal var name (pin_peer_name) stays the same to keep the
cherry-picked #27371 commits clean and avoid a noisy rename diff.
The PR #27371 resolver introduced three identity-mapping config keys
(pinPeerName, userPeerAliases, runtimePeerPrefix), but operators had
no guided way to set them — they had to read the README, understand
the resolver ladder, and hand-edit honcho.json. This commit adds an
interactive step to 'hermes honcho setup' that asks one question
('what's your deployment shape?') and writes the right combination
of keys.
Three shapes cover the realistic deployments:
* single -- pinPeerName=true. All gateway users collapse to your
peerName. Recommended for personal/single-operator use.
* multi -- pinPeerName=false, no aliases. Each runtime user gets
their own peer. Optional runtimePeerPrefix for cross-
platform namespace isolation.
* hybrid -- pinPeerName=false, with userPeerAliases mapping YOUR
runtime IDs (Telegram UID, Discord snowflake, Slack
user, Matrix MXID) to peerName. Multi-user gateway
where you are a privileged operator.
A 'skip' option leaves existing identity-mapping config untouched —
critical because re-running setup must not silently wipe operator-
curated aliases.
The wizard detects the current shape from existing config so the
prompt's default matches what the operator already has.
PR #27371 introduced a per-user-peer resolver in HonchoSessionManager,
but the resolved runtime identity is frozen into the manager at first-
message init. When the gateway session_key intentionally omits the
participant ID (the default for threads via thread_sessions_per_user=
False), a cached AIAgent created by user A is reused for user B's
messages, attributing B's writes to A's resolved Honcho peer and
breaking #27371's per-user-peer contract.
Fix by including user_id and user_id_alt in _agent_config_signature so
the cache key distinguishes participants in shared threads. Each user
in a shared thread now triggers a fresh AIAgent build (trading prompt-
cache warmth for memory-attribution correctness — the right tradeoff
for an external-memory backend where misattribution is unrecoverable).
The default-None case keeps the signature byte-identical to pre-fix
behavior so this change doesn't invalidate in-flight caches on deploy.
PR #27371 added host-scoped userPeerAliases, runtimePeerPrefix, and
pinPeerName, but the cloned-profile allowlist in
plugins/memory/honcho/cli.py::clone_honcho_for_profile() omitted them.
A new profile created via 'hermes honcho setup' or similar would
silently drop the operator's identity-mapping config, causing gateway
users to resolve to raw runtime IDs and fragmenting Honcho memory
across an unintended set of peers.
Add the three keys to the allowlist and a regression test class
covering all three plus the unset case.
Closes#33163.
When _try_activate_fallback() switches from one provider to another (e.g.
openai-codex → openrouter), the credential pool still belongs to the
primary provider. This causes two compounding bugs:
1. The pool retains the primary's base_url. Downstream pool recovery
(rate_limit / billing / auth) calls _swap_credential() with a primary
entry which overwrites the agent's base_url back to the primary's
endpoint. Every fallback request then 404s against the wrong host.
2. Pool recovery acting on errors from the FALLBACK provider mutates the
PRIMARY's pool state (#33088 reported a related corruption pattern),
exhausting/rotating entries that have nothing to do with the failure.
Two layered fixes:
a) try_activate_fallback (agent/chat_completion_helpers.py): on fallback
activation, clear agent._credential_pool when the fallback provider
doesn't match the pool's provider. Pool is preserved when the fallback
shares the pool's provider (e.g. multiple openrouter entries).
b) recover_with_credential_pool (agent/agent_runtime_helpers.py):
defensive guard rejects any pool mutation when agent.provider doesn't
match pool.provider. Defense-in-depth — should never fire after (a)
is in place, but covers any future path that attaches a stale pool.
Salvaged from @zccyman's PR #33217. The original PR was written against
the pre-refactor monolithic run_agent.py; both target functions have
since been extracted to module-level helpers. Behavior is identical —
the guards live in the canonical extracted locations.
Tests
- New tests/run_agent/test_fallback_credential_isolation.py (7 tests
covering: fallback clears mismatched pool, fallback preserves matching
pool, recovery rejects mismatched pool, recovery accepts matching
pool, 429-from-z.ai-doesn't-exhaust-codex-pool, _client_kwargs
base_url survives pool clear, _swap_credential doesn't restore
primary URL after fallback).
- Cross-verified: 77/77 passing across fallback isolation tests +
agent/test_credential_pool.py — no regression.
Co-authored-by: zccyman <16263913+zccyman@users.noreply.github.com>
In profile mode, _load_provider_state previously returned None when a
provider was absent from the profile's auth.json — even if the user had
authenticated at the global root. This broke runtime credential resolvers
that read state directly (resolve_nous_access_token,
resolve_nous_runtime_credentials), causing profiles without their own
nous login to fail with 'Hermes is not logged into Nous Portal' despite
a valid global session.
Push the existing read-only global fallback (already used by
get_provider_auth_state and read_credential_pool) into _load_provider_state
so every caller benefits, and simplify get_provider_auth_state into a thin
wrapper. Writes still target the profile only — profile state continues to
shadow global state on the next read after a per-profile login. Behavior in
classic (non-profile) mode is unchanged because _load_global_auth_store
returns an empty dict.
Adds 5 tests covering the new contract on _load_provider_state directly.
Existing 770 auth/credential/nous tests still pass.
Closes#33175.
switch_model() in agent/agent_runtime_helpers.py mutated agent.model and
agent.provider before rebuilding the client, with no try/except to restore
them on failure. If the rebuild raised (bad API key, network error,
build_anthropic_client failure, etc.) the agent was left with the new
model+provider name paired with the OLD client — producing HTTP 400s like
"claude-sonnet-4-6 is not supported on openai-codex" on the next turn.
Callers in cli.py, gateway/run.py, and tui_gateway/server.py already catch
the exception and warn the user, but the warning was misleading because
the swap had partially succeeded; the agent's state was torn.
Snapshot every mutated field before the swap, wrap the swap+rebuild block
in try/except, and restore the snapshot on failure before re-raising so
the caller's warning surfaces.
Reported by @amirariff91. Tests cover both branches (chat_completions and
anthropic_messages) and the cross-branch case (anthropic -> openai).
Three additions on top of @Nami4D's salvage:
1. Gate the preflight slash-enum strip on the model name pattern
(grok-* / x-ai/grok-*). The original PR stripped slash-containing
enum values from every codex_responses request, but native Codex
(OpenAI) and GitHub Models DO accept slash enums — stripping them
there would silently degrade tool-schema constraints. xAI is the
only Responses-API surface that rejects the shape.
2. Resolve the merge conflict in agent/transports/codex.py by
preserving both the timeout-forwarding block that landed on main
between the PR's branch point and now AND the new service_tier
strip. Behavioural intent of both is preserved.
3. Six new tests in tests/agent/transports/test_codex_transport.py
covering:
- TestCodexTransportXaiServiceTierStrip (3 tests): xAI strips
service_tier from request_overrides; non-xAI codex_responses
and GitHub Models both KEEP service_tier (regression guards
so the strip stays xAI-only).
- TestPreflightSlashEnumStrip (3 tests): Grok and aggregator-
prefixed Grok model names both trigger the safety-net strip;
non-Grok models preserve slash enums as a regression guard
against the strip becoming too broad.
51/51 in tests/agent/transports/test_codex_transport.py.
Co-authored-by: Nami4D <hello@nami4d.tech>
#33151 flipped THREE Telegram display defaults to false:
- tool_progress: new -> off (kept: per-tool stream is too chatty)
- interim_assistant_messages: T -> F (REVERTED here)
- long_running_notifications: T -> F (REVERTED here)
- busy_ack_detail: T -> F (kept: verbose iteration counter)
The two reverts were wrong. interim_assistant_messages = the model's REAL
words mid-turn ("I'll inspect the repo first.", "Let me check both files
in parallel"). That is signal, not noise. Suppressing it left Telegram
users staring at "typing..." for the entire turn duration with no
feedback. long_running_notifications = the periodic heartbeat. Silent
agent for 30 minutes is worse than one bubble updating every 3 minutes.
Changes:
- gateway/display_config.py: Telegram tier-1 inbox keeps both defaults
on (only tool_progress and busy_ack_detail stay off).
- gateway/run.py _notify_long_running(): edit a single heartbeat
message in place (where the adapter supports it) instead of posting
a new "Still working..." bubble each interval. Telegram, Discord,
Slack, Matrix all qualify. Falls back to send-new when edit fails.
- gateway/run.py: tighten heartbeat text. "⏳ Still working... (12 min
elapsed — iteration 21/60, running: terminal)" -> "⏳ Working — 12
min, terminal". Verbose iteration detail moves behind busy_ack_detail
(one knob now controls both busy acks AND heartbeat verbosity).
- tests/, cli-config.yaml.example, website/docs/user-guide/messaging:
updated to reflect the corrected story.
Closes#32992.
The chat path resolves Codex credentials via `resolve_codex_runtime_credentials`
which only reads `providers.openai-codex.tokens` (the singleton). The auxiliary
path uses `_read_codex_access_token` which checks the credential_pool first.
For users whose tokens live only in the pool — manual seed, partial re-auth,
restore from backup, or any state where the singleton is empty but the pool
is healthy — the chat path raised AuthError or (worse, since OpenAI(api_key='')
silently attaches no header) the wire saw HTTP 401 "Missing Authentication header"
while the auxiliary path worked fine.
This adds a pool fallback to `resolve_codex_runtime_credentials`: when the
singleton has no usable access_token, scan `credential_pool.openai-codex` for
the first entry that has a non-empty access_token and isn't in an exhaustion
cooldown window (`last_error_reset_at` in the future). If found, return that
token with `source="credential_pool"`. If no usable entry exists, the original
AuthError propagates as before.
Regression tests cover:
- Empty singleton + healthy pool entry → pool token returned
- Pool fallback skips entries currently in cooldown
- Empty singleton + empty/wedged pool → AuthError propagates (existing contract preserved)
Two tests in test_verbose_command.py asserted Telegram's tool_progress
default was "new" and expected /verbose to cycle that to "all". The
default has since been overridden to "off" in gateway/display_config.py
(_PLATFORM_DEFAULTS for telegram — tier-1 inbox preset that keeps mobile
chats final-answer-first), making the first /verbose invocation cycle
off → new, not all → verbose.
The behavioral change was intentional; the tests were stale and missing
from the same commit. Surfaced as a pre-existing failure on origin/main
during CI for the unrelated #33164 / #33168 Codex auth salvages.
When the Codex OAuth token endpoint returns 429 (usage-limit / quota
exhaustion), refresh_codex_oauth_pure raised a generic auth error that the
gateway surfaced as 'Primary provider auth failed: No Codex credentials
stored. Run hermes auth', prompting re-auth that cannot lift a quota cap.
Classify 429 distinctly (codex_rate_limited, relogin_required=False) with a
non-alarming quota message that honors Retry-After, log it as
'Primary provider rate-limited (429)', and stop format_auth_error from
appending the re-authenticate remediation. Also log the fallback provider's
literal config key instead of the resolved runtime category.
Refs #32790
Codex re-auth via `hermes setup` / `hermes model` wrote fresh OAuth
tokens to providers.openai-codex.tokens but left the credential_pool
device_code entry holding the consumed refresh token and stale error
markers. Since the runtime selects from the pool, the next request
spent a dead token and got a 401 token_invalidated. Update the
singleton-seeded pool entries in lockstep and clear their error state.
Fixes#33000
reasoning.encrypted_content is sealed to the Responses endpoint that
minted it. When a session switches model providers mid-conversation —
say the user runs /model gpt-5.5 after several turns on grok-4.3, or
vice versa — the persisted codex_reasoning_items carry blobs the new
endpoint cannot decrypt, and every subsequent turn fails with HTTP 400
invalid_encrypted_content.
This is the cross-issuer prevention layer. Pairs with:
* PR #33035 — runtime recovery when the HTTP 400 fires anyway
* PR #33146 — prevention for transient rs_tmp_* items
Stamps each reasoning item with the issuer kind that minted it
(codex_backend / xai_responses / github_responses / other:<url>) at
normalize time, then drops items at replay time when the active
endpoint differs from the stamp. Unstamped (legacy) items pass
through for backwards compatibility.
Cherry-picked from @chaconne67's PR #31629. Conflict against current
main (#33035's replay_encrypted_reasoning parameter) resolved as
'keep both' — the two guards compose: replay_encrypted_reasoning=False
is the session-wide kill switch, current_issuer_kind is the per-item
filter that runs only when replay is still enabled.
Background processes whose command contains `gh pr view --json
statusCheckRollup` or `gh pr checks | jq` now get a runtime hint in
the result pointing at the canonical green-ci-policy snippets. The
homebrew shape has caused at least seven silent CI-watcher failures
in the past two weeks (#31329, #31448, #31695, #31709, #31745,
#32264, #33131) — each one a different jq/awk/grep variation of the
same fundamental problem (stdout buffering, jq null-key edge cases,
conclusion-vs-status confusion, TTY-only banner grepping).
The skill that documents this anti-pattern is excellent, but a skill
only fires if the agent loads it. The tool surface fires on every
misuse. This is the embed-footguns-in-tool-surface pattern from
PR #31289 applied to a recurring failure mode that's outgrown
skill-only enforcement.
Detector is deliberately narrow — flags two specific shapes:
1. Any command containing `statusCheckRollup` (the JSON-API path —
conclusion vs status field semantics keep burning us).
2. `gh pr view` / `gh pr checks` combined with `jq` (gh pr
checks doesn't emit JSON, so any `| jq` here is confused intent;
the canonical column-2 poller uses awk-on-tabs, not jq).
Does NOT flag the blessed column-2 awk-on-tabs poller (which uses
`awk -F"\t" "\==\"pending\""`) or the exit-code-driven
`gh pr checks $PR >/dev/null` snippet.
Hint composes with the existing background-without-notify_on_complete
hint — both can fire on the same call. Each is independently
actionable.
Tests:
- 4 new cases in tests/tools/test_notify_on_complete.py
- test_homebrew_ci_poller_via_statusCheckRollup_emits_hint (positive)
- test_homebrew_ci_poller_via_gh_pr_checks_piped_to_jq_emits_hint (positive)
- test_canonical_column2_awk_poller_does_not_emit_homebrew_hint (negative)
- test_canonical_gh_pr_checks_exit_code_loop_does_not_emit_hint (negative)
- test_non_ci_background_command_does_not_emit_homebrew_hint (negative)
- 30/30 passing (was 26)
Operators behind reverse proxies that don't reliably forward
X-Forwarded-Host / X-Forwarded-Proto / X-Forwarded-Prefix (manual
nginx setups, on-prem ingresses, custom-domain Fly deploys with
incomplete proxy chains) had no way to force the absolute base URL
the OAuth callback redirects from. The dashboard would reconstruct
the redirect_uri from request headers, the IDP would echo it back,
and the user would land on the wrong host or wrong path — 404.
Add `dashboard.public_url` to config.yaml with env override
HERMES_DASHBOARD_PUBLIC_URL. When set, it is the complete authority —
scheme + host + optional path prefix (e.g. https://example.com/hermes) —
and becomes the base for the OAuth `redirect_uri`. X-Forwarded-Prefix
is IGNORED on this code path because the operator has explicitly
declared the public URL; we no longer need to guess from proxy
headers, and stacking the prefix on top would double-prefix the
common case where the prefix is already baked into public_url.
When unset, the existing proxy_headers + X-Forwarded-Prefix
reconstruction runs untouched. Existing Fly.io deploys continue to
work without configuration — this is purely additive.
Precedence mirrors dashboard.oauth.client_id:
env (non-empty) > config.yaml > reconstructed from request
Implementation:
- hermes_cli/config.py: add dashboard.public_url to DEFAULT_CONFIG
with a multi-paragraph doc comment explaining the use case,
the X-Forwarded-Prefix interaction, and the validation rules.
- hermes_cli/dashboard_auth/prefix.py: factored out the existing
_REJECT_CHARS frozenset, added _normalise_public_url() validator
(requires http/https scheme + non-empty host + no header-injection
chars), _load_dashboard_section() loader (robust to load_config
raising, non-dict shapes), and resolve_public_url() entry point
with the env-overrides-config precedence. A malformed value
silently falls through to ""; the caller treats "" as "reconstruct
from request" so a typo never breaks the login flow.
- hermes_cli/dashboard_auth/routes.py: rewrite _redirect_uri()
docstring to spell out the three resolution tiers; add the
public_url short-circuit before the existing X-Forwarded-Prefix
splicing. Source-level comment notes that X-Forwarded-Prefix is
intentionally ignored when public_url is set so a future reader
doesn't try to "fix" the missing prefix layering.
- cli-config.yaml.example: extend the existing dashboard section
with a public_url block.
- website/docs/user-guide/features/web-dashboard.md: new "Public
URL override" section between the provider configuration and
the OAuth flow walkthrough. Documents the env-vs-config table,
the validation rules, and the `http://` `public_url` ↔ Secure
cookie footgun.
Test coverage — new TestPublicUrlOverride class (8 tests):
- env var overrides request reconstruction (the primary motivating
case)
- config.yaml used when env unset
- env wins over config (precedence pin)
- public_url with a path prefix already baked in (the Q1-a case the
user explicitly chose)
- public_url suppresses X-Forwarded-Prefix layering (defends
against the double-prefix bug)
- trailing slash stripped from public_url (no //auth/callback)
- malformed public_url falls through to reconstruction (six
hostile inputs: javascript:, ftp:, missing scheme, missing host,
quote chars, CRLF injection)
- empty env string doesn't shadow config.yaml entry (CI / Fly
provisioned-but-empty secret case)
Mutation-tested: flipping the precedence in resolve_public_url() trips
exactly test_env_overrides_config_public_url; weakening the validator
(accept any scheme) trips exactly test_malformed_public_url_falls_through_to_reconstruction.
Both other tests in each pair stay green, confirming the suite
discriminates the specific regression each test pins.
Per AGENTS.md, ~/.hermes/.env is reserved for API keys / secrets and
config.yaml is the surface for non-secret configuration. The Nous
Portal plugin previously read HERMES_DASHBOARD_OAUTH_CLIENT_ID and
HERMES_DASHBOARD_PORTAL_URL from the environment only, which forced
local-dev / on-prem operators to put non-secret per-instance
configuration in .env — violating the convention.
Add dashboard.oauth.{client_id,portal_url} to DEFAULT_CONFIG and have
the plugin resolve each setting with env-overrides-config precedence:
1. Env var when set to a non-empty value (Fly.io platform-secret
injection — what pushes per-deploy client_ids without baking
them into the image).
2. config.yaml entry (canonical surface for local dev / on-prem).
3. Plugin default (no provider registered when client_id is empty;
portal_url defaults to https://portal.nousresearch.com).
Empty env values are explicitly treated as unset so a provisioned-but-
not-populated Fly secret can't accidentally shadow a valid config.yaml
entry with an empty string — operators would otherwise lose the gate.
Implementation:
- hermes_cli/config.py: add dashboard.oauth.{client_id,portal_url}
block to DEFAULT_CONFIG with full doc comment explaining the
override precedence and Fly.io rationale.
- plugins/dashboard_auth/nous/__init__.py: add _load_config_oauth_section,
_resolve_client_id, _resolve_portal_url helpers; replace the two
direct os.environ.get() calls in register() with the resolvers.
Update the skip-reason string to mention BOTH surfaces so an
operator looking at the fail-closed bind error knows config.yaml
is a valid alternative to the env var.
- plugins/dashboard_auth/nous/plugin.yaml: update description to
name both surfaces. requires_env stays pointing at the env var
name — it's metadata-only (not used by the plugin loader for
gating) so this is documentation/UX, not enforcement.
- cli-config.yaml.example: append commented dashboard.oauth block
with the same override rationale operators see in code.
- website/docs/user-guide/features/web-dashboard.md: rewrite the
'Default provider: Nous Research' section to lead with config.yaml,
present env vars as operator overrides (Fly.io's primary path).
Updated the example fail-closed bind error to match the new
skip-reason text.
Test coverage — new TestConfigYamlSource class (8 tests) pinning
every tier of the precedence chain:
- config-yaml-only path registers correctly
- both config-yaml fields (client_id + portal_url) honoured
- env var overrides config for client_id (Fly.io critical path)
- env var overrides config for portal_url
- empty env string does NOT shadow config (CI/Fly edge case)
- neither source set → skip with reason mentioning BOTH surfaces
- load_config() raising falls through to env-only path (resilience)
- non-dict oauth section falls through cleanly (typo resilience)
Mutation-tested: flipping the precedence to config-wins-over-env trips
exactly test_env_overrides_config_client_id while the other 7 stay
green, confirming the suite discriminates the order, not just the
sources.
This closes the last item in Teknium's PR review (PR #30156).
Mission-control style deploys reverse-proxy the dashboard at a path
prefix (e.g. mission-control.tilos.com/hermes/* -> :9119) and inject
X-Forwarded-Prefix: /hermes on every request. The SPA mount already
honoured this for asset URLs and the bootstrap __HERMES_BASE_PATH__,
but the OAuth gate didn't:
1. The gate's Location: header to /login and the 401 envelope's
login_url were built bare ("/login?next=..."). Under a /hermes
prefix the browser follows that to mission-control.tilos.com/login
which the proxy doesn't route to the dashboard.
2. _redirect_uri (the OAuth callback URL handed to the IDP) used
request.url_for() which doesn't honour X-Forwarded-Prefix
(Starlette/uvicorn only proxy_headers Host + Proto + For). The
IDP redirects back to /auth/callback instead of /hermes/auth/
callback → 404 in the user's browser.
3. Cookies were set with Path=/ which leaks them to other apps on
the same origin and won't be sent back on requests under the
prefix in the first place.
Fix threads the normalised prefix through every boundary:
* New hermes_cli/dashboard_auth/prefix.py — single source of truth
for X-Forwarded-Prefix parsing. web_server._normalise_prefix
becomes a re-export so the SPA mount, the gate, and the cookies
helper all agree.
* middleware._unauth_response builds login_url = f"{prefix}/login".
* routes._redirect_uri splices the prefix into the path component
of the IDP-bound URL (with full validation of the header).
* cookies.{set,clear}_{session,pkce}_cookie now take prefix="".
Path attribute switches to /hermes when set; cookie name switches
name variant (see below). Every caller passes the request's
normalised prefix.
Cookie hardening (Teknium's lesser-note #1 in the PR review): adopt
the __Host- / __Secure- cookie name prefixes per draft-west-cookie-
prefixes. The variant is selected from (use_https, prefix):
* Loopback HTTP → bare "hermes_session_at" (both prefixes require
Secure, incompatible with HTTP).
* HTTPS, direct deploy (Path=/) → "__Host-hermes_session_at".
Strongest spec: bound to exact origin, no Domain attribute, Secure
required.
* HTTPS, behind a proxy prefix (Path=/hermes) →
"__Secure-hermes_session_at". __Host- forbids Path != "/"; the
explicit Path=/hermes covers same-origin app isolation.
Setter and reader BOTH consult the prefix because the cookie *name*
changes — a reader that looked up the bare name when the setter wrote
__Secure- would never find the value. The reader falls back across
all three variants so a request whose shape changed mid-session (e.g.
post-deploy from no-prefix to /hermes) still picks up the existing
cookie until it expires.
Test coverage:
- tests/hermes_cli/test_dashboard_auth_prefix.py — new file. 11 tests
pinning:
• Location: /hermes/login on the gate's HTML redirect
• 401 envelope login_url carries the prefix
• Malformed X-Forwarded-Prefix is ignored (header-injection
defence; the script-tag value is normalised to empty string)
• _redirect_uri splices /hermes into the path (the property
that prevents the IDP-returns-to-404 failure)
• PKCE cookie uses Path=/hermes + __Secure- when proxied
• Session cookies use __Host- when direct, __Secure- when
proxied, bare on loopback HTTP
• End-to-end round trip with hand-managed PKCE cookie carriage
(TestClient can't simulate a Path=/hermes cookie automatically)
- tests/hermes_cli/test_dashboard_auth_cookies.py — rewritten to pin
each (use_https, prefix) shape produces its expected cookie name,
plus reader-side coverage that __Host- and __Secure- variants are
both recognised.
- Existing tests across middleware / 401-reauth / etc. updated to
match the new cookie names (substring contains instead of
startswith).
Mutation-tested: reverting _unauth_response to build the bare
"/login" URL trips exactly the two tests that pin the prefix
carriage, confirming the suite discriminates the regression.
