force_close_tcp_sockets stayed shutdown-only after #29507 to avoid
cross-thread FD recycle. That left CLOSED sockets unreclaimed when
httpx.close() skipped already-shutdown sockets under long-lived
gateways (~1 CLOSED fd / 6 min via proxy).
Add release_fds= for the owning-thread dispose path only; abort still
defaults to shutdown-only.
Only restore held verification text when the loop genuinely ends through budget exhaustion. Preserve later interrupts and failures, keep generated-summary fragment explanations, and add regression coverage for both contracts.
Treat intent-ack continuation text as non-final so last-turn exhaustion requests a real summary instead of surfacing a premature promise. Keep iteration-limit fallback text free of the abnormal-fragment explainer.
Track held-back verification responses explicitly so budget exhaustion returns the composed report without a second model call. Keep unrelated error and recovery exit reasons intact, preserve Kanban timeout accounting, and cover the real run_conversation paths.
* test: deflake CI and dev-machine flaky tests in bulk
Fixes ten distinct flake sources found by mining recent CI failures and
running the full suite on a dev machine with real user state:
CI-observed races:
- tests/conftest.py live-system guard: allow signal 0 (pure liveness
probe) through _guarded_kill/_guarded_killpg. psutil.pid_exists()
probes a just-killed grandchild reparented to init; the subtree check
fails for it and the guard RuntimeError'd
test_entire_tree_is_sigkilled_not_just_parent intermittently on
unrelated PRs.
Hermeticity flakes (fail on dev machines with real state, pass on CI):
- agent/coding_context.py: _marker_root() now skips the shared temp
root (tempfile.gettempdir()) like it skips $HOME — a stray
/tmp/package.json flipped every tmp_path test into the coding
posture (9 failures in test_coding_context.py).
- test_agent_guardrails.py: pin MAX_CONCURRENT_CHILDREN=3 via autouse
monkeypatch instead of freezing the user's real config value at
import time (import-time vs call-time config mismatch).
- test_web_tools_config.py: TestCheckWebApiKey now neutralizes the
ddgs package probe and registry providers — the optional ddgs
package in a dev venv lit up the fallback backend.
- test_credential_pool.py: block claude_code/hermes-oauth credential
autodiscovery in the two pool-merge tests that assert exact id
lists (a real ~/.claude/.credentials.json seeded an extra entry).
- test_modal_sandbox_fixes.py: clear _permanent_approved /
_session_approved — the user's real command_allowlist silently
approved the guard-escalation commands under test.
- test_setup_irc.py: stub prompt_checklist to select only the IRC row;
the non-TTY cancel fallback re-ran the real configured platforms'
interactive setup_fn, which hit input() under captured stdin.
- test_doctor.py: TestGitHubTokenCheck now patches the module-level
HERMES_HOME constant (the file's established pattern) instead of
only setenv — doctor was running PRAGMA integrity_check against the
real multi-GB state.db and blowing the 300s per-file budget.
Latent atexit-duplication (same _enter_buffered_busy class as #34217):
- test_undo_command.py: drop importlib.reload(tui_gateway.server) in
fixture teardown; reload re-registers the module's atexit hooks.
- test_session_platform_resolution.py: drop per-test reload of
tui_gateway.server; every resolver reads env at call time.
* test: sentinel model value in ignore-user-config fallback assertion
With HERMES_IGNORE_USER_CONFIG=1, load_cli_config() falls back to the
repo-root cli-config.yaml (untracked, gitignored). On a dev machine that
file can legitimately set the same popular model the test hardcoded
(anthropic/claude-sonnet-4.6), flipping the != assertion locally while
CI (no cli-config.yaml) stayed green. Use an impossible sentinel model
name instead.
Three tests for the #61099 salvage: OpenRouter attribution headers
present after switching to openrouter.ai, Kimi User-Agent sentinel
present after switching to api.kimi.com, and stale headers cleared
when switching to a provider with no URL-specific headers.
2/3 fail on unpatched main (DID NOT ATTACH), confirming the bug.
switch_model() unconditionally set agent.provider but only set
agent.base_url when the resolved value was truthy. When a real
provider change resolved an empty base_url (e.g. minimax after
copilot), the agent ended up with provider="minimax" but
base_url still pointing at api.githubcopilot.com. That incoherent
pair then got snapshotted into agent._primary_runtime, so it kept
re-applying on every subsequent turn via restore_primary_runtime()
until the process restarted.
try_activate_fallback() and _swap_credential() were audited and
confirmed unaffected: both always derive base_url from an actually
constructed client, never from a possibly-empty resolver hint.
Fix: when base_url is empty AND the provider is genuinely changing,
raise ValueError instead of silently keeping the old provider's URL.
This routes through switch_model()'s existing snapshot/rollback
path, and callers (tui_gateway/server.py's _apply_model_switch)
already catch and surface a clean "switch failed, staying on X"
message. Re-selecting the SAME provider with an empty base_url
(credential-only refresh) still keeps the current URL, unchanged.
Fixes#47828
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)
Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
behavior change reverting deliberate e8c3ac2f5; belongs in its own
PR with its own justification if pursued.
Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
Review finding: the substring check ('kimi' or 'moonshot' in model)
under-matches bare release slugs like k2-thinking that the repo's
canonical _model_name_is_kimi_family matcher (anthropic_adapter.py)
already covers. Reuse it instead of a second ad-hoc matcher; add a
regression test for the bare-slug case.
Adapted from PR #26014's test file to the canonical seam
(tests/run_agent/test_anthropic_prompt_cache_policy.py _make_agent
helper) instead of a new top-level file with sys.path manipulation.
Covers: kimi-k2.6 + moonshot-v1 on OpenRouter (envelope layout),
kimi via Nous Portal, and the non-OpenRouter negative case.
- Return the boundary snapshot from
_launch_session_boundary_memory_flush as a local value instead of
staging it on self._session_boundary_snapshot. The instance-attr
handoff could leak (no memory manager configured) or mis-fire a
stale snapshot on a later /new if an exception hit between staging
and consumption. A local variable eliminates the class; the helper
also returns None when no memory manager is configured so
new_session takes the inline-switch path.
- Drop the now-dead session_id kwarg from commit_memory_session:
after the redesign no production caller passes it (gateway, TUI,
compression all use the default), and speculative params are
rejected per AGENTS.md. The explicit-old-session need is served by
cli.py's direct engine call + commit_session_boundary_async.
- Drop the dead providers snapshot in commit_session_boundary_async
(only the emptiness check used it).
- Tests updated accordingly (dead-kwarg test removed, snapshot
assertion now covered by return-value contract).
Phase-2 gates: 2a tests/cli 1048 passed + 6 memory files 137 passed;
2b programmatic live smoke 0.38ms non-blocking caller, end→switch→sync
ordering verified; 2c structured 4-angle review — no Criticals, these
warnings fixed.
When an API error carries an httpx.Response whose body was consumed via
iter_bytes() during streaming error handling (e.g. GeminiAPIError from
agent/gemini_native_adapter.py), accessing .text raises
httpx.ResponseNotRead. The secondary exception replaced the real,
already-computed provider error (429 free-tier quota guidance) with the
generic 'Attempted to access streaming response content' message on
every turn.
Guard the .text access so it degrades to an empty snippet and falls
through to the str(error) fallback, which carries the full original
message. Mirrors the existing guards in
agent/error_classifier.py::_extract_error_body() and
agent/gemini_native_adapter.py::gemini_http_error().
Fixes#59769
Salvaged from PR #59868 (guard + regression test); the unrelated
desktop Ctrl-C fix bundled in that PR was intentionally dropped and is
triaged separately.
Sessions on sub-512K-context models were spending most of their wall-clock
re-summarizing: the 50% trigger left too little post-compaction headroom
(the incompressible floor — system prompt, tool schemas, protected tail,
rolling summary — ate most of the reclaimed space), so compaction re-fired
every 1-2 turns. Three compounding defects fixed:
- Threshold floor: models with context windows below 512K now trigger at
>=75% of the window (raise-only — a higher configured value or per-model
autoraise like Codex gpt-5.5's 85% always wins). Re-derived on
update_model() in both directions.
- No max_tokens on the summary call: the summary budget is prompt guidance
only ("Target ~N tokens"). The wire cap truncated summaries mid-section
on the Anthropic Messages / NVIDIA NIM paths (thinking models burn the
cap on reasoning first), yielding truncated or thinking-only summaries
and compaction loops. Summary token ceiling lowered 12K -> 10K to keep
the guidance within the intended 1K-10K envelope.
- Reasoning traces excluded end-to-end: inline <think>/<reasoning> blocks
are now stripped from assistant content before serialization to the
summarizer, and from the summarizer's own output before the summary is
stored (previously a thinking summarizer model's trace was persisted in
_previous_summary and re-fed into every iterative update, compounding
bloat). Native reasoning fields were already excluded.
Verified E2E with real imports against a temp HERMES_HOME: threshold table
across 64K-1M windows, override interactions (user 0.85 wins, spark 0.70
raised, gpt-5.5 0.85 kept), full compress() round-trip with a thinking
summarizer, and wire-kwargs capture proving no max_tokens is sent.
Review findings on the salvaged #60332 breaker, fixed as follow-ups:
- restore_primary_runtime() now resets the streak (third provider-swap
path; without it a recovered primary was short-circuited before a
single attempt and could never be re-proven healthy except via /model).
- interruptible_api_call (non-streaming) now carries the same breaker
(guard at entry, bump on stale_call_kill, reset on success). Quiet-mode
/ subagent / headless sessions — the profile most like #58962's
unattended 494-failure session — take this path and had the identical
infinite stale-retry class.
- Partial-stream stub return now resets the streak (chunks were received,
provider demonstrably responsive).
- Consolidated the triple-duplicated counter arithmetic into shared
helpers (_stale_streak/_bump_stale_streak/_reset_stale_streak/
_check_stale_giveup) with one canonical comment block; error message
now says 'consecutive stale attempts' (the counter counts kills, not
turns — a single turn can produce several).
4 new tests (restore resets / no-op restore keeps latch / non-streaming
short-circuit / non-streaming success reset).
Follow-up for the salvaged #60332 circuit breaker. The breaker latches:
once the streak trips, interruptible_streaming_api_call raises before any
stream is attempted, so the on-success reset can never run again. The
error text tells the user to switch models and retry — but neither
switch_model() nor try_activate_fallback() cleared the streak, so a
freshly selected healthy provider kept short-circuiting forever (only
/new recovered), and the automatic fallback chain was wedged the same way.
Reset the streak at both swap sites (after a successful rebuild only;
rollback/exhaustion paths keep the latch). 4 tests.
A session wedged against an unresponsive OpenAI-compatible provider can hit the stale-stream detector on every turn and loop forever, burning the full 180s x retries each turn with no response. Issue #58962 reports 494 consecutive failures over 3+ days on a single session.
The streaming retry path already caps retries WITHIN a turn (HERMES_STREAM_RETRIES, default 2) but has no cross-turn cap. Once a session's conversation state makes every turn stale, it retries indefinitely across turns and never notifies the user.
Add a per-session consecutive-stale-stream counter on the agent:
- incremented on every stale-stream kill in the outer poll loop;
- reset to 0 only when a stream actually completes;
- when it reaches HERMES_STREAM_STALE_GIVEUP (default 5), the next turn aborts immediately with a clear, actionable RuntimeError instead of spending 180s x retries again.
This is distinct from the existing stale-stream work (local-provider hard ceiling #44938, backoff/parse-error #60031): those bound a single hung stream, while this bounds repeated cross-turn staleness and surfaces a user-visible error.
Adds tests/run_agent/test_stream_stale_circuit_breaker.py covering the short-circuit, the success-reset, and the increment.
Drop the Responses-API native compaction path and its opt-in umbrella
flag from the salvaged feature. On the Codex OAuth chat route Hermes
owns the message list and the summary compressor works (and stays
provider-portable — encrypted compaction items would lock the session
history to chatgpt.com and break /model switches and provider
fallback). On the app-server runtime (codex CLI/agent) the codex agent
owns the real thread context, so thread/compact/start is the only
mechanism that can actually shrink it (#36801) — that path is now the
default behavior for codex_app_server sessions, controlled by
compression.codex_app_server_auto (native|hermes|off), no umbrella
flag.
Removed: responses.compact() call path, codex_compaction_items replay/
persistence plumbing, codex_native_compaction + codex_responses_threshold
config keys, desktop settings fields, and their tests. Kept: everything
app-server (compact_thread(), compaction notifications, bookkeeping,
docs, tests) plus cache-busting keys for the surviving knobs.
The provider-mismatch guard now checks pool_provider and
current_provider != pool_provider. MagicMock.provider returns
a truthy child mock by default, which would trigger the guard
and skip the pool recovery tests. Set pool.provider='' explicitly.
When context.engine selects a plugin engine (e.g. LCM), the host
compression threshold — including the Codex gpt-5.5 50% -> 85%
autoraise — only configures the built-in ContextCompressor and never
reaches the plugin. The autoraise notice still fired, telling the user
auto-compaction was raised when nothing actually changed, and the
startup context-limit line printed the host percent next to the
engine's own threshold_tokens, contradicting itself.
- Clear _compression_threshold_autoraised when a plugin engine is
selected, suppressing both the CLI startup notice and the gateway
turn-1 replay via _compression_warning.
- Print the active engine's own threshold_percent in the startup
context-limit line so percent and token count agree.
- Built-in behavior is preserved, including the fallback path where a
configured engine fails to load and the built-in compressor takes
over.
Fixes#44439
gpt-5.3-codex-spark has a native 128K context window but the default
50% compaction trigger fires at ~64K, wasting half the usable window
before the session has accumulated enough turns to summarize
meaningfully. This raises the trigger to 70% (~90K) on the Codex OAuth
route only, leaving ~38K headroom for the summary and continued
conversation before the 128K hard limit.
The override is not gated by allow_codex_gpt55_autoraise because 128K
is the model's native window (unlike gpt-5.5's artificial 272K Codex
cap). Non-Codex routes are unaffected.
Also adds a boundary regression test verifying the short-session
scenario from the issue always yields a non-empty compressible window
(no silent context wipe).
ruff check --fix --select F541 . on current main. Pure prefix removals;
adjacent-string concatenations keep the f only on interpolating fragments.
No string content or live placeholder altered.
TestMcpParallelToolBatch seeded provenance under old-style
mcp_<server>_<tool> names, which no longer pass the
is_mcp_tool_parallel_safe() prefix gate after the naming change.
coerce_tool_args only repaired the outermost value, so JSON-encoded
*elements* of array properties (and nested object sub-fields) were left
as strings. Three core tools have array<object> schemas — todo.todos,
delegate_task.tasks, memory.operations — so a model emitting
{"todos": ["{...}"]} would pass raw JSON strings into the tool and fail
downstream on item["id"]/item["goal"] access.
Adds a schema-guided recursive pass (_normalize_json_strings_for_schema)
that parses JSON-string array items and nested object fields only when
the matching schema position expects an array/object, preserving
legitimate JSON-looking string fields (type: string).
Adapted from cline/cline#11803 to hermes-agent's existing coercion layer.
Follow-up to the salvaged #58696 (devatnull) + #41343 (annguyenNous)
commits: instead of fully suppressing commentary/analysis-phase stream
deltas, fire on_reasoning_delta so the CLI/gateway display them like
thinking text. Matches Codex CLI semantics where commentary is never
the turn's final answer, while keeping the narration visible in the
reasoning display. Adds devatnull to AUTHOR_MAP.
The custom ``httpx.HTTPTransport(socket_options=[SO_KEEPALIVE, ...])``
in ``_build_keepalive_http_client()`` was introduced to fix CLOSE-WAIT
socket accumulation on long-lived connections (#10324).
That approach broke streaming for providers behind reverse proxies
(OpenResty, Cloudflare, etc.) because the custom socket options
conflict with the proxy's chunked-transfer handling (#54049, #12952).
It also stripped TCP_NODELAY, stalling TLS handshakes and SSE encoding.
Narrow per-provider bypasses were added for Copilot (#50298), Codex
(#36623, #12953), but the root cause remained.
The fix moves connection lifecycle management from the socket layer to
the HTTP pool layer:
- ``httpx.Limits(keepalive_expiry=20.0)`` tells httpx to close idle
pooled connections at 20 s, before a reverse proxy's typical 30-60 s
timeout drops them and causes CLOSE-WAIT accumulation.
- The default httpx transport preserves OS TCP defaults (including
TCP_NODELAY), so TLS handshakes and SSE chunked encoding work
correctly.
- ``trust_env=False`` prevents httpx from double-dipping on env vars
(we handle proxy detection ourselves via ``_get_proxy_for_base_url``
which respects NO_PROXY).
- The Copilot host bypass (line 3632) is no longer needed since all
providers now use the same standard httpx.Client.
Closes#54049. Supersedes #12010, #36623, #12953, #50298.
Extend the pre_tool_call plugin hook return contract with a new directive:
{"action": "approve", "message": "why this needs human confirmation"}
Previously a pre_tool_call hook could only veto a tool call (action: block)
or allow it silently. It could not escalate to the existing human-approval
flow. This unlocks user-defined runtime approval rules on ANY tool (HTTP
writes, file writes to sensitive paths, email sends), enforced at runtime —
resolving #51221 as a pure plugin, with no core approval.py rule schema.
Mechanism:
- get_pre_tool_call_directive() returns (action, message) for block|approve;
get_pre_tool_call_block_message() kept as a block-only back-compat shim.
- resolve_pre_tool_block() is the single dispatch-site chokepoint: fetches
the directive and, for approve, invokes the human gate; fail-closed to a
block on denial, timeout, or gate exception. ALL FOUR tool-dispatch sites
now call it: tool_executor (concurrent + sequential), agent_runtime_helpers,
and model_tools.handle_function_call.
- request_tool_approval() escalates via the SAME machinery as Tier-2
dangerous commands: session/permanent allowlist, prompt_dangerous_approval
(CLI) / submit_pending (gateway), [o]nce/[s]ession/[a]lways/[d]eny,
timeout fail-closed, approvals.cron_mode for cron contexts.
Architecture: extracted the shared decision core into _run_approval_gate(),
called by BOTH check_dangerous_command() and request_tool_approval() so the
fail-closed / cron / gateway / yolo / persist policy lives in ONE place and
cannot drift. Fixed a latent divergence — the plugin path now honors --yolo.
Approval grain: [a]lways is keyed on tool_name + a hash of the reason (an
explicit plugin rule_key overrides), so distinct reasons on the same tool
persist independently instead of one 'always' blanketing the whole tool.
Non-interactive: cron honors approvals.cron_mode (parity with commands); any
other non-interactive non-gateway context fails CLOSED for the plugin path
(the command path keeps its historical fail-open default, unchanged).
No new config schema, no new env vars, no new hook events.
Strict providers (DeepSeek) reject a payload where the same tool_call_id
appears more than once with HTTP 400 'Duplicate value for tool_call_id'.
The issue was filed as an 'orphaned tool message' compression bug, but the
pasted error is a DUPLICATE tool_call_id — orphans are already handled on
main; duplicates were not. Reproduced live on main: both shapes leaked
through repair_message_sequence and sanitize_api_messages.
Two chokepoints, two shapes:
- repair_message_sequence: consume the id from known_tool_ids on first
match so a SECOND tool result reusing it falls into the drop branch
(duplicate tool-result shape). This is @Robinlovelace's kernel from
#55436 (applied manually — that PR was ~800 commits stale and bundled
an unrelated duplicate-DB-write change for #860, which is dropped here).
- sanitize_api_messages (final pre-API pass): add a dedup pass covering
BOTH (a) duplicate tool_calls sharing an id WITHIN one assistant message
(the message[6] shape) and (b) later tool result messages reusing an
already-seen id. #55436 covered neither of these at this chokepoint.
Tests: duplicate-tool-result dedup at both functions, duplicate-assistant-
tool_call-id collapse, and a negative control proving distinct ids are
never dropped (no over-dedup).
Credit: @Robinlovelace (#55436) for the repair_message_sequence dedup kernel.
Closes#58327.
repair_message_sequence Pass 1 registered only tc.get("id") when building
the set of known assistant tool_call ids, then matched tool results against
it by tool_call_id. In the Codex Responses format an assistant tool_call
carries both id (fc_...) and a distinct call_id (call_...); a tool result's
tool_call_id may be keyed on either depending on which builder produced it.
Registering only id made a valid tool result whose tool_call_id matched
call_id look orphaned, so the pass dropped it and left the assistant
tool_call unanswered -- producing HTTP 400 on strict providers (DeepSeek,
Kimi): 'Messages with role tool must be a response to a preceding message
with tool_calls'. Long-running sessions that persisted such a sequence were
permanently broken, re-sending the orphan every turn.
Register both id and call_id for each assistant tool_call so a result
matching either key is recognized, consistent with
AIAgent._get_tool_call_id_static and the compressor's _sanitize_tool_pairs.
Apply the same call_id||id precedence to the corrupted-args sanitizer's
existing-result scan / stub insertion, which had the identical mismatch.
Adds 3 regression tests covering the codex id!=call_id case (match on
call_id, match on only call_id, match on id when both present).
The salvaged max-output/pressure fix set conversation_history=None after
the new pre-API compaction. That is only correct for legacy session-
rotation. Under the default in-place compaction (compression.in_place:
True), archive_and_compact inserts the compacted rows into the session DB
directly without stamping them with the intrinsic persisted-marker, so a
subsequent flush with conversation_history=None re-appends them — doubling
the active context and retriggering compression (the early-persist
duplicate-row trap).
Use conversation_history_after_compression(agent, messages), matching the
two existing compaction sites (post-response should_compress and the
turn-prologue preflight), which returns None for rotation and
list(messages) for in-place so the compacted dicts are skipped by identity.
Adds a regression test with a real SessionDB + real archive_and_compact
that asserts the compacted summary row is persisted exactly once (fails
with 2 copies on the None variant).
Follow-up to the per-site strips from the review gate. The two copy-site
strips are correct but positional — a copy site added after the assembly
loops would re-leak _db_persisted into the child-session flush. Add a single
terminal sweep (_strip_persistence_markers) run once on the fully-assembled
compressed list so the invariant 'no compacted message leaves compress()
carrying a persistence marker' is structural, not dependent on copy-site order.
- agent/context_compressor.py: _strip_persistence_markers() called before
compress() returns; helper docstring notes the sweep is the authoritative guard
- tests/agent/test_context_compressor.py: structural regression — neuter the
per-site helper to a leaking copy, assert the terminal sweep still strips
- tests/run_agent/test_compression_persistence.py: pin the fixture assumption
behind the exact-equality row-count assertion
Shallow messages[i].copy() during context compression propagated the
_db_persisted marker from cached gateway incremental flushes into the
post-rotation compressed list. _flush_messages_to_session_db then skipped
every row when writing to the new child session, so gateway restarts
lost the compacted transcript (severe amnesia).
Strip the marker in _fresh_compaction_message_copy() and add regression
tests for rotation flush + compressor assembly.
Fixes#57491
