Add a bounded turn-end stop guard for kanban workers. When a worker
tries to exit with finish_reason=stop without having called
kanban_complete or kanban_block, inject up to two synthetic nudges
so the conversation loop continues instead of exiting cleanly (which
the dispatcher records as protocol_violation).
Mirrors the existing verify-on-stop pattern: same ephemeral scaffolding
flag (_kanban_stop_synthetic), same role-alternation contract, same
_pending_verification_response fallback for budget exhaustion.
Disabled by default (gated on HERMES_KANBAN_TASK env var set by the
dispatcher); kill switch via HERMES_KANBAN_STOP_NUDGE=0.
Salvaged from #62262 by @mdc2122. The original branch was 272 commits
behind main with ~538 files of stale-base reversions; this salvage
applies only the 4 substantive files (agent/kanban_stop.py,
conversation_loop.py insertion, run_agent.py _EPHEMERAL_SCAFFOLDING_FLAGS,
tests/agent/test_kanban_stop.py).
Salvage of #63888. The original fix clears stale _last_content_with_tools
on substantive tool-only turns but doesn't clear _mute_post_response, which
a prior housekeeping turn may have set. This suppresses tool progress
output via _vprint until the no-tool-call branch resets it at line ~4834
— after all tools have finished executing.
Fix: also reset _mute_post_response = False when clearing stale fallback.
Added test: verify pure housekeeping turns (content + only housekeeping
tools) still set the fallback correctly — the original use case the
fallback was designed for.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
A cached _last_content_with_tools response from a housekeeping-only turn
could survive a later substantive tool-only turn. When the model returned
an empty response, Hermes incorrectly finalized the older housekeeping
narration instead of invoking the post-tool empty-response nudge.
Production impact: scheduled cron jobs could return early without completing
their actual work (e.g., daily report job returning a housekeeping message
instead of producing the report artifact).
Root cause: The fallback state was only updated when a turn had both
content AND tool_calls. A turn with tool_calls but empty visible content
would skip state updates entirely, leaving stale fallback state intact.
Fix: Classify tools in every tool-call turn (regardless of visible content).
When any tool is substantive (non-housekeeping), clear the older fallback state
before processing later empty responses. This prevents two-turn-old housekeeping
narration from being treated as if it belonged to the immediately preceding
substantive tool turn.
Regression test added: tests/run_agent/test_conversation_fallback_state.py
Fixes#63860
OpenAI lets ChatGPT-plan Codex users bank rate-limit reset credits, but
until now they could only be redeemed from the Codex CLI/app or the
website. This wires the same backend API into Hermes:
- /usage on the openai-codex provider now shows "You have N resets
banked - use /usage reset to activate" (parsed from the
rate_limit_reset_credits field the /usage endpoint already returns).
- New /usage reset subcommand (CLI + gateway) redeems one banked
credit via POST .../rate-limit-reset-credits/consume with a UUID
idempotency key, mirroring codex-rs backend-client semantics
(PathStyle /wham vs /api/codex, ChatGPT-Account-Id header,
reset/nothing_to_reset/no_credit/already_redeemed outcomes).
- Guard: redemption is refused while no rate-limit window is fully
exhausted, since a banked reset restores the FULL 5h + weekly
allowance and spending it early wastes it. /usage reset --force
overrides. Zero banked credits and non-codex providers are refused
with clear messages; nothing_to_reset reports the credit was NOT
spent.
- i18n: new gateway.usage.unknown_subcommand / reset_wrong_provider
keys across all 16 locales; docs updated (cli.md, messaging index).
Tested with unit tests plus a real-socket E2E against a local fake
Codex backend exercising redeem/guard/force and the /usage hint.
Cache-decorated turns (apply_anthropic_cache_control converts string
content to [{type: text, ..., cache_control}] lists — applied BEFORE the
MoA facade since the #57675 cache-cold fix) and multimodal turns
(text + image_url parts) flattened to empty strings in
_reference_messages, which only read str content. On turn 1 of a
provider:moa session with a Claude aggregator the references received a
single EMPTY user message: Anthropic-side providers 400'd ('messages: at
least one message is required') while tolerant models answered 'no user
request is present' (live incident Jul 14 2026, preset 'closed').
Fixes, in totality:
- _reference_messages: extract visible text via
agent/message_content.flatten_message_text for user/assistant/tool
turns (skips image parts, so no base64 leaks into the advisory view);
decorated and undecorated transcripts now produce a byte-identical
advisory view (advisor cache prefix stays stable).
- image-only user turns get a placeholder instead of an empty message
(Anthropic rejects empty text blocks) or a silently dropped turn
(would break user/assistant alternation).
- degenerate-case fallback flattens structured content too.
- _attach_reference_guidance: a decorated/multimodal trailing user turn
now receives the guidance as a NEW text part appended AFTER the
cache_control-marked part (cached prefix byte-stable) instead of
falling through to a second consecutive user message (strict providers
reject user/user).
- conversation_loop MoA injection: multimodal user turns get the MoA
context appended as a trailing text part instead of being dropped;
user_prompt for the one-shot path flattens content lists instead of
str()-ing them (which leaked base64 payloads into the prompt).
Live-verified on the 'closed' preset (real OpenRouter wire, 2 user
turns, tool loop): all 4 reference calls carry the full document +
rendered tool state, end on user, zero tool-role/tool_calls; advisor
cache_write 7968 then cache_read 5909+; aggregator cache_read
14880-15237 on iterations 2+.
Co-authored-by: bo.fu <bo.fu@meituan.com>
test_probe_sends_client_context_to_gemini and
test_probe_omits_gemini_client_context_for_other_providers (added in
b8eb89f5c) patch hermes_cli.models.urllib.request.urlopen, but
probe_api_models routes requests through the
_urlopen_model_catalog_request wrapper (open_credentialed_url from the
urllib_security hardening), so the mock is never invoked and
mock_urlopen.call_args is None -> TypeError. Every CI run on main and
every PR has been failing test slice 7/8 on these two tests.
Point the patches at _urlopen_model_catalog_request, the same target
every sibling test in TestProbeApiModelsUserAgent already uses.
89/89 tests in the file now pass.
Include the Hermes client name and version with Gemini inference, model and tier checks, and TTS requests. Add focused coverage for the request headers and keep the Gemini-specific context scoped to Google Gemini endpoints.
The desktop provider panel previously rendered the curated declarations
from hermes_cli/memory_providers.py: five hindsight fields, and no panel
at all for undeclared providers like honcho (OAuth connect only). The
dashboard provider-switching rework re-pointed the shared config route
at raw plugin schemas, so the desktop began dumping every internal field
(35 for hindsight) and grew a bespoke honcho panel.
Serve both surfaces from the same route: ?surface=declared returns the
curated schema (empty for undeclared providers) with the original
config-file + env-store write semantics; the dashboard keeps the raw
plugin schema unchanged. The desktop client opts into declared.
The dashboard's dedicated memory-provider UI (4b184cbe5) excluded
memory.provider from /api/config/schema server-side. Desktop's settings
page builds its field list from that schema, so the Memory Provider
dropdown silently vanished from Desktop after v0.18.1.
- web_server.py: restore memory.provider as a select in _SCHEMA_OVERRIDES,
with options built from plugins.memory discovery (was a stale hardcoded
[builtin, honcho] list before the removal)
- plugins/memory: add list_memory_provider_names() — directory-scan-only
name listing, safe at module import time (no provider imports)
- web ConfigPage: hide memory.provider client-side instead — the Plugins
page owns the dedicated provider-switching UI there
- tests: schema contract (select present, category memory, builtin
sentinel) + invariant that every discoverable provider is selectable
#63955 made Hermes survive a broken `bash -l` (Ainz's `Directory
\drivers\etc does not exist`) by falling back to non-login `bash -c`.
But a non-login shell never sources /etc/profile, so it never gets
`…\usr\bin` on PATH — and that dir holds every coreutil the file/terminal
tools shell out to (cat, mktemp, mv, wc, head, stat, chmod, mkdir, find).
Result: `write_file` returned bytes_written:0 with an EMPTY error (the
failure text went to a missing binary's stderr) and terminal commands
exited 127. The survive-broken-login-bash fix was only half-done: it
stopped crashing but silently failed every write.
Derive Git Bash's bin dirs (mingw64/bin, usr/bin, bin, …) from the
resolved bash.exe and prepend them to the subprocess PATH on Windows, in
/etc/profile precedence order so coreutils win over same-named System32
tools (find.exe, sort.exe) inside the shell. No-op off Windows and when a
login snapshot is healthy (the snapshot re-exports the full PATH inside
the shell), so this only bites on the broken-login fallback path.
Adds _git_bash_bin_dirs() (derivation, cached) + _prepend_git_bash_dirs()
(PATH merge), plus regression tests for PortableGit/MinGit layouts and
the run-env injection ordering.
Salvage of #63862. is_output_cap_error() returns False for vLLM/LM Studio
error messages that contain 'prompt contains ... input tokens' (treated as
input-overflow signal). But parse_available_output_tokens_from_error() CAN
extract a valid available_tokens from those same messages. The
compression-disabled guard only checked is_output_cap_error(), so vLLM/LM
Studio users with compression off still got a terminal failure instead of
the max-tokens retry.
Fix: also exempt when parse_available_output_tokens_from_error() returns a
value — that function determines whether the retry path can actually handle
the error, so it's the right predicate for the exemption.
Added test: verify vLLM-format error with compression_disabled=False still
triggers the max-tokens retry path.
Co-authored-by: dmabry <dmabry@users.noreply.github.com>
The retry loop computed safe_out from the error's available_tokens,
which reflected the *previous* request. Between retries the agent
appends tool results and error text, so the real input token count
grows. Deriving safe_out from the stale budget meant every retry
still exceeded the context ceiling by 1+ tokens, burning through the
3-attempt limit.
Compute safe_out from estimate_messages_tokens_rough(messages) so
the cap tracks the growing input on each retry attempt.
Salvage of #63935. The original fix read HERMES_CRON_SESSION_DB_TIMEOUT
from a bare env var, but AGENTS.md requires non-secret behavioral
settings to live in config.yaml with an env var bridge only for
backward compatibility.
Changes:
- Add cron.session_db_timeout_seconds to DEFAULT_CONFIG (default 10s)
- Resolution order: HERMES_CRON_SESSION_DB_TIMEOUT env override →
cron.session_db_timeout_seconds in config.yaml → 10s default
(mirrors the existing script_timeout_seconds pattern)
- 0 = unlimited (opt-in for debugging, skips the bound)
- Strengthen test: assert the warning is logged on invalid env value
(caplog was taken but never asserted)
- Add test: verify config.yaml resolution path works end-to-end
Co-authored-by: LoicHmh <26006141+LoicHmh@users.noreply.github.com>
run_job() constructs SessionDB() synchronously with no timeout of its
own, unlike the agent's run_conversation call further down, which is
already bounded by HERMES_CRON_TIMEOUT. A wedged sqlite3.connect (e.g.
a stale flock from a crashed sibling process) hangs this call
indefinitely.
That hang is invisible to every existing cron safeguard because it
happens before _submit_with_guard's future exists: the finally block
that discards the job ID from _running_job_ids never runs. The job
stays wedged "running" — every later tick logs "already running —
skipping" — until the whole gateway process is restarted.
Observed in production: a cron job's worker thread was confirmed via
a live py-spy thread dump to be parked inside SessionDB.__init__'s
sqlite3.connect for 3+ days, silently skipping every scheduled fire
in between across a gateway process that otherwise stayed healthy.
Bound the SessionDB() construction with its own timeout
(HERMES_CRON_SESSION_DB_TIMEOUT, default 10s), following the same
bounded-thread-pool pattern already used elsewhere in this file (the
delivery retry path, and the agent inactivity watchdog just below).
On timeout, log at ERROR and proceed with session_db=None instead of
degrading silently to debug level, since an actual hang here is a new
condition worth surfacing.
Adds tests/cron/test_sessiondb_init_hang.py, including an end-to-end
regression proving the dispatch guard is released and a subsequent
tick can fire the same job again after a simulated hang.
Same fix as the base-env bootstrap tests: init_session's post-failure
non-login probe was overwriting captured["script"] with `true`. Capture
the first (bootstrap) _run_bash call via setdefault.
init_session now fires a follow-up non-login `bash -c true` probe when the
login bootstrap fails, so the two bootstrap-script assertions must capture
the first call only (setdefault), not the probe that overwrote it.
Commit 62a76bd3d (feat: make smart approvals the default, #62661)
changed approvals.mode default from "manual" to "smart". The
TestBlockingApprovalE2E tests did not patch the approval mode, so
check_all_command_guards routed through _smart_approve() first —
calling call_llm() which tried all auxiliary providers, failed (no
API keys in test env), and returned "escalate" before falling through
to the gateway blocking path. The LLM failure cascade took longer
than the test's 2.5s wait window (50 × 0.05s), so the notify
callback had not fired yet when assert len(notified) == 1 ran.
Force {"mode": "manual"} via patch in setup_method/teardown_method,
matching the pattern already used by test_blocking_approval_uses_
canonical_timeout in the same class.
#63621 fixed path quoting, but Ainz's Git for Windows still dies on
`bash -l` itself (`Directory \drivers\etc`). Hermes then fell back to
bash -l *per command*, so every write_file/terminal call failed the same way.
After a failed login snapshot, probe non-login bash -c; if it works, skip
-l for the session. Also skip a stale HERMES_GIT_BASH_PATH that fails a
noprofile probe in favor of %LOCALAPPDATA%\hermes\git portable bash.
Splits the single broad `except Exception` in the compression-lock
acquire path into two handlers: AttributeError/TypeError (version skew —
the lock method is missing, or predates the `ttl_seconds=` kwarg) still
fails OPEN as before, since that's known-safe to proceed without a lock.
Any other exception now fails CLOSED (skips compression this cycle)
instead of treating every failure as "no lock subsystem present" and
letting a second compressor run concurrently and fork the session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
Closes#63529
Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.
Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.
Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
Normalize Anthropic setup-token metadata for every PooledCredential construction path, persist corrected manual entries, heal legacy rows on load without copying global fallback credentials into profiles, and map the contributor email for release attribution.
Reject metadata that would make session queries fail, bound import work, and detach cyclic lineage links. Guard lineage traversal against pre-existing corrupt cycles.
