Provider auto-detection (URL-based inference for Anthropic, OpenAI Codex,
and xAI endpoints) runs before credential-pool validation in AIAgent init,
but #63048 placed the pool validation before auto-detection. When the agent
is constructed with provider=None and a recognized endpoint URL, the pool
is validated against an empty provider identity and discarded, even though
auto-detection correctly resolves the provider moments later.
Fix: move the credential-pool validation block to after the URL-based
auto-detection chain. The pool is stored on the agent before
auto-detection; validation now checks the resolved provider and only
nullifies agent._credential_pool when the pool's scoped provider genuinely
doesn't match.
Regression test covers all three auto-detection paths:
- Anthropic (api.anthropic.com)
- OpenAI Codex (chatgpt.com/backend-api/codex)
- xAI (api.x.ai)
Fixes#63425.
- Nudge text now warns that repeated protocol violations will block the
task and require manual intervention, so the model understands the
consequence of ignoring the nudge.
- Kanban docs restructured to clearly separate the two defense layers:
agent-side prevention (nudge, from #64350) and dispatcher-side
recovery (bounded retry, from this PR).
- Two new integration tests verifying the nudge mentions blocking and
that the agent-side and dispatcher-side budgets are independent.
Fold in kevinb361's suggested lifecycle wording (#61817 conceded in favor of
this PR) and update the second stale site his sweep didn't cover: the
task-events table still said the dispatcher 'auto-blocks immediately instead
of retrying'. Both now describe the violation-only streak: protocol_violation
fires on every violation (its payload marker feeds the budget), below-budget
runs return the task to ready, and gave_up + auto-block happen only when the
consecutive streak reaches _PROTOCOL_VIOLATION_FAILURE_LIMIT (default 3,
per-task max_retries overriding).
Address the hermes-sweeper review of #61233: the bounded retry budget
is now a clean-exit-specific streak, not a share of the unified
consecutive_failures counter.
- detect_crashed_workers stamps a protocol_violation marker into the
violation run's metadata (via the event payload _end_run copies);
_protocol_violation_streak derives the streak from run history:
consecutive most-recent violation runs, rate_limited runs neutral
(mirroring their unified-counter treatment), any other closed run
resets it. Mixed failure kinds can neither consume nor extend the
budget.
- Below-budget violations no longer call _record_task_failure at all:
the task returns to ready with last_failure_error stamped directly
(including the corrective retry guidance wording adopted from #61817,
which build_worker_context surfaces to the retry worker) and the
unified counter is untouched, keeping the two budgets independent.
- At the bound the trip funnels through _record_task_failure with a new
keyword-only force_trip=True: the reaper has already resolved the
per-task max_retries override against the violation streak itself, so
the threshold comparison is skipped rather than double-applied.
max_retries keeps its documented top precedence in both directions:
max_retries=1 blocks on the first violation, max_retries=5 blocks on
the fifth consecutive one, unset uses the default bound of 3.
- Replace the first-violation-blocks regression test with five tests:
first occurrence retries (ready + guidance stamped + no gave_up +
unified counter untouched); streak trips exactly at the bound with
protocol_violations/protocol_violation_limit in the gave_up payload
and the auto-blocked side channel set; a prior nonzero crash does not
consume the violation budget; a non-violation failure between
violations resets the streak; max_retries precedence both directions.
All five fail against the previously reviewed diff and pass with this
follow-up. The test harness resolves hermes_cli.kanban_db fresh and
uses that single module object for the exit registry, liveness patch,
and reaper — earlier suite tests reload the module, and the old
mixed-object harness made _classify_worker_exit return unknown (the
reason the old test failed in full-suite runs on main).
Kanban suite: zero introduced failures vs upstream/main tip (62 vs 63
pre-existing environmental failures — the one no longer failing is the
old violation test this replaces; 662 passed vs 657).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A worker that exits 0 without calling kanban_complete/kanban_block
(model stops early, transient tool wedge) tripped the failure breaker
on FIRST occurrence and the task was blocked. These are overwhelmingly
transient: with a bounded retry (limit 3, tracked via a violation
fingerprint) ~96%% of them complete on respawn. Genuine repeat
offenders still trip the breaker at the limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scan_skill_commands() had two collision bugs in the same loop body:
1. Core-command collision: a skill whose normalized slug matches a core
Hermes command name or alias (e.g. "skills", "learn", "bg") would
get an auto-generated /command that shadows the core command in the
gateway dispatch path (skill map is consulted before built-in
handlers). The skill command silently overrode the core command.
2. Inter-skill slug collision: the seen_names set deduped on the raw
frontmatter name, but the command map was keyed by the normalized
slug. Two distinct names collapsing to the same slug (e.g.
"git_helper" vs "git-helper") both passed the dedup, and the second
silently clobbered the first.
Fix: add two guards in scan_skill_commands() after slug normalization:
- resolve_command(cmd_name) check skips skills colliding with any core
CommandDef (name or alias), logging a warning. Uses the existing
resolve_command() API so aliases and case variants are covered
without a separate cache. The skill remains loadable via /skill.
- cmd_key in _skill_commands check dedups on the resolved slug,
first-wins (preserving local-before-external precedence), logging
a warning naming the shadowed skill.
Combines and supersedes #31204 (@cyrkstudios), #53450 (@Gridzilla),
#50304 (@petrichor-op), and #63305 (@Vissirexa).
Co-authored-by: cyrkstudios <cyrkstudios@users.noreply.github.com>
Co-authored-by: Gridzilla <Gridzilla@users.noreply.github.com>
Co-authored-by: petrichor-op <petrichor-op@users.noreply.github.com>
Co-authored-by: Vissirexa <Vissirexa@users.noreply.github.com>
synchronous=FULL issues plain fsync(), not F_FULLFSYNC. The
F_FULLFSYNC barrier comes from checkpoint_fullfsync=1, set by the
separate _apply_macos_checkpoint_barrier(). The original docstring
conflated the two PRAGMAs.
On Darwin, the default synchronous=NORMAL only calls fsync(), which Apple
explicitly states does not guarantee data-on-platter or write-ordering.
During a WAL checkpoint race with process termination (e.g., launchd
shutdown), this can leave the main DB with half-written btree pages,
resulting in btreeInitPage error 11 corruption.
WAL mode's durability guarantee assumes the OS honors fsync barriers; macOS
does not unless we explicitly set synchronous=FULL (which issues fsync() and
F_FULLFSYNC via checkpoint_fullfsync=1).
Previously, apply_wal_with_fallback() skipped setting synchronous=FULL when
the DB was already in WAL mode, leaving connections at the unsafe
synchronous=NORMAL default. This commit adds _enforce_macos_synchronous_full()
to always enforce synchronous=FULL on macOS after any WAL activation.
Fixes#63531
Add a bounded turn-end stop guard for kanban workers. When a worker
tries to exit with finish_reason=stop without having called
kanban_complete or kanban_block, inject up to two synthetic nudges
so the conversation loop continues instead of exiting cleanly (which
the dispatcher records as protocol_violation).
Mirrors the existing verify-on-stop pattern: same ephemeral scaffolding
flag (_kanban_stop_synthetic), same role-alternation contract, same
_pending_verification_response fallback for budget exhaustion.
Disabled by default (gated on HERMES_KANBAN_TASK env var set by the
dispatcher); kill switch via HERMES_KANBAN_STOP_NUDGE=0.
Salvaged from #62262 by @mdc2122. The original branch was 272 commits
behind main with ~538 files of stale-base reversions; this salvage
applies only the 4 substantive files (agent/kanban_stop.py,
conversation_loop.py insertion, run_agent.py _EPHEMERAL_SCAFFOLDING_FLAGS,
tests/agent/test_kanban_stop.py).
Salvage of #63888. The original fix clears stale _last_content_with_tools
on substantive tool-only turns but doesn't clear _mute_post_response, which
a prior housekeeping turn may have set. This suppresses tool progress
output via _vprint until the no-tool-call branch resets it at line ~4834
— after all tools have finished executing.
Fix: also reset _mute_post_response = False when clearing stale fallback.
Added test: verify pure housekeeping turns (content + only housekeeping
tools) still set the fallback correctly — the original use case the
fallback was designed for.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
A cached _last_content_with_tools response from a housekeeping-only turn
could survive a later substantive tool-only turn. When the model returned
an empty response, Hermes incorrectly finalized the older housekeeping
narration instead of invoking the post-tool empty-response nudge.
Production impact: scheduled cron jobs could return early without completing
their actual work (e.g., daily report job returning a housekeeping message
instead of producing the report artifact).
Root cause: The fallback state was only updated when a turn had both
content AND tool_calls. A turn with tool_calls but empty visible content
would skip state updates entirely, leaving stale fallback state intact.
Fix: Classify tools in every tool-call turn (regardless of visible content).
When any tool is substantive (non-housekeeping), clear the older fallback state
before processing later empty responses. This prevents two-turn-old housekeeping
narration from being treated as if it belonged to the immediately preceding
substantive tool turn.
Regression test added: tests/run_agent/test_conversation_fallback_state.py
Fixes#63860
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
Cache-decorated turns (apply_anthropic_cache_control converts string
content to [{type: text, ..., cache_control}] lists — applied BEFORE the
MoA facade since the #57675 cache-cold fix) and multimodal turns
(text + image_url parts) flattened to empty strings in
_reference_messages, which only read str content. On turn 1 of a
provider:moa session with a Claude aggregator the references received a
single EMPTY user message: Anthropic-side providers 400'd ('messages: at
least one message is required') while tolerant models answered 'no user
request is present' (live incident Jul 14 2026, preset 'closed').
Fixes, in totality:
- _reference_messages: extract visible text via
agent/message_content.flatten_message_text for user/assistant/tool
turns (skips image parts, so no base64 leaks into the advisory view);
decorated and undecorated transcripts now produce a byte-identical
advisory view (advisor cache prefix stays stable).
- image-only user turns get a placeholder instead of an empty message
(Anthropic rejects empty text blocks) or a silently dropped turn
(would break user/assistant alternation).
- degenerate-case fallback flattens structured content too.
- _attach_reference_guidance: a decorated/multimodal trailing user turn
now receives the guidance as a NEW text part appended AFTER the
cache_control-marked part (cached prefix byte-stable) instead of
falling through to a second consecutive user message (strict providers
reject user/user).
- conversation_loop MoA injection: multimodal user turns get the MoA
context appended as a trailing text part instead of being dropped;
user_prompt for the one-shot path flattens content lists instead of
str()-ing them (which leaked base64 payloads into the prompt).
Live-verified on the 'closed' preset (real OpenRouter wire, 2 user
turns, tool loop): all 4 reference calls carry the full document +
rendered tool state, end on user, zero tool-role/tool_calls; advisor
cache_write 7968 then cache_read 5909+; aggregator cache_read
14880-15237 on iterations 2+.
Co-authored-by: bo.fu <bo.fu@meituan.com>
test_probe_sends_client_context_to_gemini and
test_probe_omits_gemini_client_context_for_other_providers (added in
b8eb89f5c) patch hermes_cli.models.urllib.request.urlopen, but
probe_api_models routes requests through the
_urlopen_model_catalog_request wrapper (open_credentialed_url from the
urllib_security hardening), so the mock is never invoked and
mock_urlopen.call_args is None -> TypeError. Every CI run on main and
every PR has been failing test slice 7/8 on these two tests.
Point the patches at _urlopen_model_catalog_request, the same target
every sibling test in TestProbeApiModelsUserAgent already uses.
89/89 tests in the file now pass.
Include the Hermes client name and version with Gemini inference, model and tier checks, and TTS requests. Add focused coverage for the request headers and keep the Gemini-specific context scoped to Google Gemini endpoints.
The desktop provider panel previously rendered the curated declarations
from hermes_cli/memory_providers.py: five hindsight fields, and no panel
at all for undeclared providers like honcho (OAuth connect only). The
dashboard provider-switching rework re-pointed the shared config route
at raw plugin schemas, so the desktop began dumping every internal field
(35 for hindsight) and grew a bespoke honcho panel.
Serve both surfaces from the same route: ?surface=declared returns the
curated schema (empty for undeclared providers) with the original
config-file + env-store write semantics; the dashboard keeps the raw
plugin schema unchanged. The desktop client opts into declared.
The dashboard's dedicated memory-provider UI (4b184cbe5) excluded
memory.provider from /api/config/schema server-side. Desktop's settings
page builds its field list from that schema, so the Memory Provider
dropdown silently vanished from Desktop after v0.18.1.
- web_server.py: restore memory.provider as a select in _SCHEMA_OVERRIDES,
with options built from plugins.memory discovery (was a stale hardcoded
[builtin, honcho] list before the removal)
- plugins/memory: add list_memory_provider_names() — directory-scan-only
name listing, safe at module import time (no provider imports)
- web ConfigPage: hide memory.provider client-side instead — the Plugins
page owns the dedicated provider-switching UI there
- tests: schema contract (select present, category memory, builtin
sentinel) + invariant that every discoverable provider is selectable
#63955 made Hermes survive a broken `bash -l` (Ainz's `Directory
\drivers\etc does not exist`) by falling back to non-login `bash -c`.
But a non-login shell never sources /etc/profile, so it never gets
`…\usr\bin` on PATH — and that dir holds every coreutil the file/terminal
tools shell out to (cat, mktemp, mv, wc, head, stat, chmod, mkdir, find).
Result: `write_file` returned bytes_written:0 with an EMPTY error (the
failure text went to a missing binary's stderr) and terminal commands
exited 127. The survive-broken-login-bash fix was only half-done: it
stopped crashing but silently failed every write.
Derive Git Bash's bin dirs (mingw64/bin, usr/bin, bin, …) from the
resolved bash.exe and prepend them to the subprocess PATH on Windows, in
/etc/profile precedence order so coreutils win over same-named System32
tools (find.exe, sort.exe) inside the shell. No-op off Windows and when a
login snapshot is healthy (the snapshot re-exports the full PATH inside
the shell), so this only bites on the broken-login fallback path.
Adds _git_bash_bin_dirs() (derivation, cached) + _prepend_git_bash_dirs()
(PATH merge), plus regression tests for PortableGit/MinGit layouts and
the run-env injection ordering.
Salvage of #63862. is_output_cap_error() returns False for vLLM/LM Studio
error messages that contain 'prompt contains ... input tokens' (treated as
input-overflow signal). But parse_available_output_tokens_from_error() CAN
extract a valid available_tokens from those same messages. The
compression-disabled guard only checked is_output_cap_error(), so vLLM/LM
Studio users with compression off still got a terminal failure instead of
the max-tokens retry.
Fix: also exempt when parse_available_output_tokens_from_error() returns a
value — that function determines whether the retry path can actually handle
the error, so it's the right predicate for the exemption.
Added test: verify vLLM-format error with compression_disabled=False still
triggers the max-tokens retry path.
Co-authored-by: dmabry <dmabry@users.noreply.github.com>
The branch computed safe_out from estimate_messages_tokens_rough(messages),
but the provider rejected the larger api_messages request (system prompt,
injected context, tool schemas). When API-only content is large, safe_out
could far exceed the provider's available_tokens.
Compute safe_out from estimate_request_tokens_rough(api_messages, tools=...)
and keep provider available_out as an upper bound. Do not alter context_length
or trigger compression for output-cap errors.
Add production-path run_conversation tests that assert the retry API call's
max_tokens, including a case where a large system prompt makes messages-only
estimation undercount the real request.
Fixes#55546
The retry loop computed safe_out from the error's available_tokens,
which reflected the *previous* request. Between retries the agent
appends tool results and error text, so the real input token count
grows. Deriving safe_out from the stale budget meant every retry
still exceeded the context ceiling by 1+ tokens, burning through the
3-attempt limit.
Compute safe_out from estimate_messages_tokens_rough(messages) so
the cap tracks the growing input on each retry attempt.
`_module_registers_tools()` reads each `tools/*.py` file and fully
AST-parses it to check for a top-level `registry.register()` call.
90 files are scanned on every process start — but only 32 actually
register tools.
Add a cheap text prefilter: after reading the file (which we need to
do anyway for AST), check that both `"registry"` and `"register"`
appear in the source before calling `ast.parse`. A file with a
top-level `registry.register()` call must contain both strings, so
this is a perfect superset — zero false negatives. 50 of 90 files
skip the AST parse entirely.
The `source=` parameter is not threaded through `discover_builtin_tools`;
the prefilter lives entirely inside `_module_registers_tools`, keeping
the public API unchanged.
Benchmark (median of 10 runs, scanning 90 files):
before (read + ast.parse all): 305.9ms
after (text prefilter + ast): 187.8ms
speedup: 1.6x (118ms saved)
Identical module set: 32 modules, same names, same order.
The _PROVIDER_PREFIXES frozenset in agent/model_metadata.py is static
and does not auto-extend from ProviderProfile. Removing deepinfra and
deep-infra from it broke provider:model prefix stripping for DeepInfra.
Preserve one durable staged input across terminal close and the worker's early turn flush, without duplicating resumed transcripts or creating a session with a null prompt. Fixes#63766.
Retain a distinct CLI history baseline during the signal window before a turn's normal persistence flush. When CLI history aliases the live agent list, use marker-only persistence so a genuinely unflushed tail is written.
Salvage of #63935. The original fix read HERMES_CRON_SESSION_DB_TIMEOUT
from a bare env var, but AGENTS.md requires non-secret behavioral
settings to live in config.yaml with an env var bridge only for
backward compatibility.
Changes:
- Add cron.session_db_timeout_seconds to DEFAULT_CONFIG (default 10s)
- Resolution order: HERMES_CRON_SESSION_DB_TIMEOUT env override →
cron.session_db_timeout_seconds in config.yaml → 10s default
(mirrors the existing script_timeout_seconds pattern)
- 0 = unlimited (opt-in for debugging, skips the bound)
- Strengthen test: assert the warning is logged on invalid env value
(caplog was taken but never asserted)
- Add test: verify config.yaml resolution path works end-to-end
Co-authored-by: LoicHmh <26006141+LoicHmh@users.noreply.github.com>
run_job() constructs SessionDB() synchronously with no timeout of its
own, unlike the agent's run_conversation call further down, which is
already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g.
a stale flock from a crashed sibling process) hangs this call
indefinitely.
That hang is invisible to every existing cron safeguard because it
happens before _submit_with_guard's future exists: the finally block
that discards the job ID from _running_job_ids never runs. The job
stays wedged "running" — every later tick logs "already running —
skipping" — until the whole gateway process is restarted.
Observed in production: a cron job's worker thread was confirmed via
a live py-spy thread dump to be parked inside SessionDB.__init__'s
sqlite3.connect for 3+ days, silently skipping every scheduled fire
in between across a gateway process that otherwise stayed healthy.
Bound the SessionDB() construction with its own timeout
(HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same
bounded-thread-pool pattern already used elsewhere in this file (the
delivery retry path, and the agent inactivity watchdog just below).
On timeout, log at ERROR and proceed with session_db=None instead of
degrading silently to debug level, since an actual hang here is a new
condition worth surfacing.
Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end
regression proving the dispatch guard is released and a subsequent
tick can fire the same job again after a simulated hang.
Baseline bars were positioned at their absolute timeline offset, which
included the baseline run's own wait time — making duration comparisons
hard since the bars were visually offset. Now each baseline bar starts
at the same left position as its corresponding current job, so the two
bars directly overlap for at-a-glance duration comparison.
Also removed the now-unused bl_t0 / bl_max / bl_jobs_timed variables
that tracked the baseline timeline position.
