In Slack threads, ordinary follow-up messages sent while a native
multi-choice clarify was pending were consumed as clarify answers —
_coerce_text_response accepted arbitrary text for any pending entry, so
the gateway text-intercept swallowed unrelated thread messages and the
user's messages appeared to be ignored.
Tighten resolve_text_response_for_session for native interactive
multi-choice prompts (buttons rendered, awaiting_text=False):
- numeric selections ("2") still resolve to the canonical choice
- exact choice-label matches (case-insensitive) still resolve
- arbitrary prose is now REJECTED (returns False) so the message
continues as a normal turn instead of vanishing into the clarify
Behavior is preserved everywhere free text is legitimately the answer:
open-ended clarifies, explicit 'Other' text-capture mode, and the base
adapter's numbered-text fallback (which flips awaiting_text at send
time).
Salvaged from PR #62042 by @liuhao1024.
Fixes#62034
Context-compaction handoff summaries (prefixed with [CONTEXT COMPACTION])
were being returned as normal bookend_start/bookend_end messages in
session_search discovery mode. A single compaction handoff could be 57K+
chars, immediately bloating a fresh session prompt to 73K+ chars from one
search hit.
Changes:
- Add _COMPACTION_PREFIXES and _is_compaction_summary() helper
- Filter compaction summaries from bookend_start and bookend_end in _discover()
- Cap bookend content to 1200 chars and window content to 4000 chars
- Add content_truncated/original_content_chars metadata when truncation occurs
- Add 6 regression tests covering prefix detection, bookend filtering,
content capping, and legacy [CONTEXT SUMMARY] prefix
Fixes#43175
Children inherit the parent's env, repo, and toolsets but were denied
execute_code ('children should reason step-by-step, not write scripts').
That forces subagents doing mechanical multi-step work (batch file
reads, fetch-N-pages loops, filter-before-context reductions) to burn
reasoning iterations one tool call at a time.
- Remove execute_code from DELEGATE_BLOCKED_TOOLS
- Stop stripping the code_execution toolset from child bundles
- No recursion risk: the sandbox bridges only the 7 SANDBOX_ALLOWED_TOOLS
(web/file/terminal) — delegate_task and execute_code itself are not
reachable from inside a sandbox script
- Update schema text, AGENTS.md, and delegation docs
- Tests: blocked-constant, strip, and child-assembly tests updated;
new test pins execute_code as intentionally unblocked
Surgical reapply of PR #37523 onto current main (the original branch
predates the SecretSource registry refactor). _build_safe_env() now
forwards env vars tagged in env_loader._SECRET_SOURCES — widened from
Bitwarden-only to any registered secret source (Bitwarden, 1Password,
plugin backends), since the provenance map is source-agnostic.
Explicit server env: config still wins; untagged secrets stay filtered.
Fixes#37499.
test_session_expired_retry_waits_for_new_session hardcoded
_server_breaker_opened_at["hindsight"] = 123.0 to simulate a circuit breaker
whose cooldown has already elapsed. But the breaker in tools/mcp_tool.py
compares that stamp against time.monotonic() (age = monotonic() - opened_at,
elapsed when age >= _CIRCUIT_BREAKER_COOLDOWN_SEC). time.monotonic()'s origin
is arbitrary and small on a freshly-booted CI container, so age worked out to
only a few seconds there (< the 60s cooldown) — the breaker stayed open, the
half-open probe never fired, and the retry returned the "unreachable" error
instead of "bank ok". It passed on long-uptime dev boxes (large monotonic)
and failed under CI, with the reported "Auto-retry available in ~Ns" drifting
run to run as the container's monotonic clock varied.
Stamp opened_at relative to the same clock the code reads
(time.monotonic() - _CIRCUIT_BREAKER_COOLDOWN_SEC - 1.0) so the cooldown is
provably elapsed regardless of the monotonic origin, exercising the intended
half-open transition deterministically.
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.
Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
Builds on trevorgordon981's #50589 (cherry-picked as the previous commit).
The #50394 cooldown reset only ran inside the async _shutdown coroutine,
which is skipped on the empty-_servers fast path — the most common state
when a server failed to connect (failed servers are never recorded in
_servers). It was also skipped when the MCP loop wasn't running.
Clear _server_connect_retry_after/_server_connect_failures on the fast
path and in a final unconditional sweep so a full shutdown/restart always
re-attempts every configured server immediately. Adds regression tests
for both paths.
Log-storm hygiene for the reconnect machinery (#65673, #66092):
- State transitions carry exactly one WARNING each:
connected → degraded (keepalive failure), degraded → parked
(budget exhausted / rapid-drop park / permanent error),
parked → connected (revival proven healthy, via
_mark_session_proven).
- Per-attempt retry logs (initial-connect attempts, connection-lost
reconnect attempts, per-cycle rebuild notices, park-wake probes)
demoted to DEBUG.
- Backoff sleeps get +/-20% uniform jitter (_jittered) so a herd of
servers that lost the same backend doesn't retry — and log — in
lockstep.
- Add module-level _unwrap_exception_group (ported from
hermes_cli/mcp_config.py, adapted): handles nested groups, prefers
non-cancellation leaves over the CancelledErrors anyio sprays across
sibling tasks, and re-raises KeyboardInterrupt/SystemExit leaves.
- Add _classify_mcp_failure(exc) -> 'permanent'|'transient'.
Permanent: auth 401/403, NonMcpEndpointError, InvalidMcpUrlError,
FileNotFoundError/ENOENT on the stdio command. Transient: everything
else (network/EOF/ClosedResource/TaskGroup drops). Permanent failures
park immediately without burning the retry ladder — every retry
against a missing binary or a revoked credential hits the same wall.
- Every log site in run() and the keepalive failure log now logs the
unwrapped root cause as 'TypeName: message', so a dead stdio pipe
says 'BrokenPipeError' instead of the opaque
'unhandled errors in a TaskGroup (1 sub-exception)' (and empty
str(exc) dead-pipe errors are no longer blank).
Harden the immediate-reconnect path from #66271:
- _reconnect_or_reraise_group re-raises when the transport TaskGroup
carries a KeyboardInterrupt / SystemExit leaf — fatal signals must
propagate to the interpreter, never be converted into a reconnect.
- Rapid-drop budget: a completed handshake alone no longer clears
_reconnect_retries/backoff on clean transport return. A session is
UNPROVEN until it demonstrates real health — survived >=1 full
keepalive interval (keepalive success path) or served >=1 successful
tool call (_mark_session_proven). Unproven clean returns are charged
against _MAX_RECONNECT_RETRIES, so a flapping transport that
handshakes fine and drops moments later still reaches the park
instead of respawning forever (#62212: 6212 spawns in 63h).
Proven sessions keep the #57604 behaviour: budget clears, transient
blips over a long-lived session never accumulate toward parking.
Streamable-HTTP / SSE MCP transports run their stream pump inside an anyio
TaskGroup. A transient stream drop (idle timeout, brief backend blip,
server-side TCP close) surfaces as a BaseExceptionGroup escaping the
transport context manager. It reached run()'s error path, which applied
exponential backoff (1s..16s) and eventually parked the server for 300s and
deregistered its tools — turning a sub-second glitch into a multi-minute
tool outage even though the POST path was still healthy.
_run_http now wraps all three transport branches (SSE, new + deprecated
Streamable-HTTP) and routes a BaseExceptionGroup through a new
_reconnect_or_reraise_group() helper that returns 'reconnect' (immediate
rebuild, no backoff/park/deregister). It re-raises instead of masking when
shutdown is in progress, when the group carries a real CancelledError
(cancellation must propagate, cf. #9930), or when no live session was
established this attempt (a connect/handshake failure that should fall
through to run()'s backoff rather than hot-loop).
Fixes#66092
Co-authored-by: 雨哥 <hanyu1212@users.noreply.github.com>
Update the Matrix reaction-seeding contract to the four-reaction default
(once/session/always/deny), add tirith-tier (session without always) and
no-session-tier cases, and assert allow_session=True in the tirith
gateway payload.
Three related messaging-approval fixes:
1. approvals.timeout default 60 -> 300. PR #63501 collapsed the gateway
wait onto the canonical approvals.timeout (previously
gateway_timeout=300), silently shrinking messaging approval windows
to 60s. Push-notification approvals routinely arrive later than a
minute; taps landed after the wait had already failed closed.
2. Stale-tap honesty: adapters resolved the approval AFTER rendering
'<checkmark> Approved by <user>' (Telegram/Discord/Slack), or ignored a zero
resolve count (WhatsApp Cloud/Feishu). A tap on an expired prompt
claimed approval while the command had already been denied. All
button paths now resolve first and render 'Approval expired -
command was not run' when nothing was waiting.
3. Mixed-warning prompts (dangerous pattern + tirith finding) now offer
Always: the persistence layer already permanently allowlists the
pattern key and downgrades the tirith key to session scope, but the
UI hid Always whenever ANY tirith warning was present. Pure-tirith
prompts still withhold Always (content findings are session-max by
design), and Smart-DENY overrides remain once-only.
_resolve_active_context_length() called get_model_context_length() with the
model id alone, so provider-enforced windows (e.g. Codex OAuth's 272K for
gpt-5.x vs the direct API's 1.05M) never reached the tool-search activation
gate — it sized against generic metadata for the same slug.
Resolve the runtime provider for the configured model and pass provider,
base_url, and api_key through. If credential resolution fails (offline, no
keys), degrade to a provider+base_url-only lookup so the static
provider-aware fallbacks still apply; explicit model.context_length keeps
short-circuiting as before (#46620). Gap flagged during review of #16735.
Follow-up to #67640: move the agent.file_safety import to module top
(stdlib-only, no circular-import concern), replace the over-broad
except Exception + logger.warning with an import sentinel plus
logger.exception so a guard failure is debuggable instead of silently
swallowed. Adds fail-closed tests asserting the diagnostic is emitted.
The live transcripts added with delegate_task write each child's events to
<hermes_home>/cache/delegation/live/<delegation_id>/. Nothing on that path is
redacted, and the rendered events are precisely the secret-bearing surfaces:
tool args, tool results and streamed assistant text.
That location is not incidental — delegate_tool.py:1620 documents cache/
delegation as "mounted read-only into remote backends", so every line lands
in a file readable from inside the sandbox.
Observed on the writer today:
tool | -> terminal(curl -H "Authorization: Bearer sk-ant-api03-...")
result | terminal ok 0.4s: OPENAI_API_KEY=sk-proj-... AWS_SECRET_ACCESS_KEY=wJalr...
The same three values through the canonical redactor:
curl -H "Authorization: Bearer ***" https://api.internal
OPENAI_API_KEY=*** AWS_SECRET_ACCESS_KEY=***
Every other sink for this data already routes through that redactor — search
results via redact_sensitive_text(file_read=True), terminal output via
redact_terminal_output — so the transcript was the one place an operator's
keys reached disk in the clear.
Redact at three points, covering every artefact the dispatch writes into that
directory:
* event() — every typed helper (assistant_text, thinking, tool_start,
tool_result, marker, finalize, the stream flush) funnels through it, so
one call covers them all and a helper added later cannot bypass it.
* the .log header — written directly rather than through event(), and a
caller can paste a key into the task text.
* manifest.json — _write_manifest serialises the same goal, and the manifest
sits in the same mounted directory, so redacting only the header would
have left the credential exposed one file over.
force=True because this is a safety boundary and must redact even with the
global toggle off. On the (impossible-in-practice) import failure the line is
withheld instead of written raw: losing a debug line costs less than writing
a live credential into a sandbox-readable file.
Key names, tool names, statuses, durations and ordinary prose are untouched,
so tail -f stays as useful as before — pinned by its own test, alongside a
whole-directory sweep asserting no file under live/<id>/ carries the raw key.
register_credential_file() takes a skill-declared relative path from
required_credential_files frontmatter and bind-mounts it read-only into the
remote sandbox the skill's own code runs in. It validates that the resolved
path stays inside HERMES_HOME — the docstring names the threat directly:
so that a malicious skill cannot declare
required_credential_files: ['../../.ssh/id_rsa'] and exfiltrate
sensitive host files into a container sandbox
Containment is the wrong boundary on its own, because HERMES_HOME is exactly
where the master credential stores live. Traversal is blocked; asking for the
keys by name is not:
skill declares mounted? agent may read it?
.env YES DENIED
auth.json YES DENIED
.anthropic_oauth.json YES DENIED
cache/bws_cache.json YES DENIED
mcp-tokens/srv.json YES DENIED
google_token.json YES allowed
../../.ssh/id_rsa no n/a
Every row marked DENIED is refused by the canonical read guard
(agent.file_safety.get_read_block_error) — the agent cannot read_file them —
yet one line of hub-installed skill frontmatter gets them bind-mounted where
that skill can cat them. .env alone is every provider API key.
Reuse the canonical deny-list as the mount bar: what the agent is forbidden
to read is not mountable either, so the mount surface cannot hand a skill
what the read surface denies it. Fails CLOSED — if the guard can't be
consulted the mount is refused rather than risked.
The module keeps doing its job: a skill still mounts its own service token
(google_token.json, skills/*), and a refused entry is reported back through
register_credential_files' missing list instead of failing the batch.
The three prior PRs here (#3946, #3951, #4316) all hardened traversal; this
closes the half that traversal validation never covered.
An orphaned pip descendant holding the probe's inherited stdout/stderr
pipe handles wedged subprocess.run's post-timeout communicate() (which
joins the pipe reader threads with NO timeout on Windows). The warm
probe thread then hung holding the module-level _CACHE_LOCK, so every
new session's prompt build blocked indefinitely (#67964).
Two layers:
- _run(): replace subprocess.run with Popen + communicate(timeout); on
TimeoutExpired kill the process TREE (taskkill /T on Windows) and
reap the direct child bounded — never re-read the pipes, so an
orphaned descendant holding them open can't block us.
- get_environment_probe_line(): the probe now runs in a single
background worker publishing via a threading.Event; callers wait at
most _PROBE_WAIT_TIMEOUT (10s) then fail open with "". After one
full timeout, later callers only peek. If the stuck worker ever
finishes, its line resumes appearing in new prompts.
Regression tests: hung probe with 4 concurrent callers returns bounded;
late recovery publishes; repeat callers skip the wait; _run() returns
promptly despite a pipe-holding descendant (real subprocess E2E).
Fixes#67964
judge_goal() returns (verdict, reason, parse_failed, wait_directive) since
the goals.py wait-directive change, but the kanban goal-mode completion gate
at tools/kanban_tools.py still unpacked 3 values. Every judge call raised
ValueError, the defensive except swallowed it, and the pre-initialized
verdict='done' let every completion through — the acceptance gate was
silently disabled.
Now unpacks all 4 values; the test mock is updated to match the real
contract. The other two judge_goal consumers (hermes_cli/goals.py) already
use 4-value unpacks.
Reported and diagnosed by @bill3wits in PR #57276; reimplemented under
project authorship because the original commit was authored under a
non-existent local identity (bash@hermes.local) that cannot be carried
into history. Also fixes#58066 (duplicate report by @Gibcity).
Follow-up for salvaged PR #39946: file_sync.py already aliases time.sleep
as _sleep specifically to avoid tests mutating the shared stdlib module
object. Apply the same convention to the rate-limit clock (_monotonic)
and point the new regression test at it.
FileSyncManager.sync() is rate-limited to once per _sync_interval via
_last_sync_time, and its docstring promises that on failure "state rolls
back so the next cycle retries everything". But the except handler also
set _last_sync_time = time.monotonic() on failure, so the next non-forced
sync() within the interval hit the rate-limit guard and returned early —
suppressing the retry the rollback had just prepared.
Because the non-forced sync() runs before every command on the SSH, Modal
and Daytona backends, a single transient upload failure (network blip,
dropped channel) left the remote with stale files for the next command
(up to _sync_interval, default 5s). Forced syncs bypass the guard, which
is why it was intermittent.
Remove the failure-path timestamp bump so the clock only advances on a
successful or no-op cycle, matching the documented contract. Add a
regression test that fails before this change and passes after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
grok-4.5 is xAI's newest release (their versioning is non-monotonic:
4.5 > 4.20) and is the model xAI's own docs use for the server-side
x_search tool. Users who explicitly pinned x_search.model keep their
choice; everyone else picks up the new default via the config
deep-merge — no _config_version bump needed.
- tools/x_search_tool.py: DEFAULT_X_SEARCH_MODEL
- hermes_cli/config.py: DEFAULT_CONFIG x_search.model + comment
- agent/reasoning_timeouts.py: 300s stale-timeout floor entry for
grok-4.5 (grok-4.20-reasoning entry kept for pinned users)
- docs: x-search.md en + zh-Hans (config sample + troubleshooting)
- tests: default-model assertion + timeout-floor positive case
* feat(delegation): live-viewable subagent transcripts for delegate_task
Each child now streams an append-only, human-readable log to
<hermes_home>/cache/delegation/live/<delegation_id>/task-<n>.log while it
runs, and the dispatch return includes the paths so the caller can tail
them immediately instead of waiting blind for the consolidated summary.
- New tools/delegation_live_log.py: LiveTranscriptWriter (per-event append
+ flush, one-line rendering with truncation, never raises into the agent
loop), wrap_progress_callback (tees the child's existing
tool_progress_callback events into the log, preserves the _flush
contract), dispatch-time creation with pre-headered files so tail -f
attaches immediately, manifest.json (goals/task count/per-task status),
and 7-day retention pruning on new dispatches.
- delegate_task: wraps each child's progress callback with the writer;
sync results and background dispatch responses gain live_transcripts
(+ hint field on dispatch); per-task result entries carry
live_transcript; transcripts finalized with exit-reason markers.
- async_delegation: dispatch_async_delegation_batch accepts an optional
delegation_id so the live/ dir name matches the returned handle; the
completion event carries live_transcripts.
- process_registry: consolidated batch-completion block references each
task's live transcript path.
- Tool schema description documents the live_transcripts return surface;
docs gain a 'Live Transcripts' section with a tail -f example.
Placement under cache/delegation means the logs are mounted read-only
into remote terminal backends for free. Side-channel only: zero changes
to message content, so prompt caching is unaffected. Transcript-OUT only
— no overlap with the subagent control surfaces of PR #66046.
* fix(delegation): label the kickoff transcript line as user — it is the child's one user message
The salvaged unit tests hand-construct completed futures to lock the
_run_on_mcp_loop contract. Add a live-loop test that reproduces the actual
field trigger: an inner asyncio.wait_for expiry stores a real TimeoutError
on a real future scheduled on the MCP loop. Asserts the fixed loop surfaces
it once, promptly -- not spinning to the outer deadline (which both leaked
memory and masked the real error behind the generic wrapper message).
Bug 1 of #55048: when the MCP connection dropped (driver crash / restart),
_lifecycle_coro exited but left _started=True, so the next list_apps/capture
passed _require_started() and then operated on a None session — hanging
forever instead of reconnecting.
- _lifecycle_coro's finally now resets _started=False on ANY exit, so a dead
session is re-enterable (idempotent no-op on the normal stop() path; atomic
bool write, safe from the bridge-loop thread without the lock stop() holds).
- call_tool() re-enters start() when the session isn't active, rebuilding it
before the call. The start_session/end_session handshake (driven by start()/
stop() themselves) is exempted so bootstrap doesn't recurse.
Tests: two cases in test_computer_use_delivery_ladder.py — finally resets
_started, and call_tool restarts a dead session exactly once. Full
computer_use suite green (233).
Refs #55048 (Bug 1). Bug 2 (expose foreground dispatch) is covered by the
delivery_mode work in #67123.
Hermes' computer_use wrapper dropped cua-driver's structured action verdicts,
exposed no delivery_mode, and injected background-only guidance — so the agent
reported unverified no-ops as success and concluded cua-driver 'cannot drive'
Electron/Chromium surfaces (observed live on tldraw offline). Fixes#67052.
Phase A — preserve the result contract:
- ActionResult carries verified/effect/escalation/path/degraded/code/delivery_mode
- CuaDriverBackend._action() reads structuredContent (was data-only); a helper
normalizes it, additive and None-safe on old drivers
- _text_response surfaces the fields additively (ok stays transport-only)
Phase B — bounded, model-reachable foreground:
- delivery_mode (background|foreground) + bring_to_front on the schema, dispatcher,
ABC, and all input methods
- foreground is capability-gated (input.delivery_mode); old drivers get a
structured foreground_unsupported refusal, never a silent background downgrade
- no automatic/hidden foreground retry — the model selects it from the signal
Phase C — guidance + isolation:
- system prompt (prompt_builder) and bundled skills/computer-use/SKILL.md go from
background-ONLY to background-FIRST, teaching the AX→PX→foreground ladder driven
by returned effect/escalation, not predicted from the app being Electron
- foreground approval scoped by (action, delivery_mode): a background approval
never silently authorizes foreground
- approval state keyed per session_id so concurrent gateway runs don't leak unlocks
Tests: tests/tools/test_computer_use_delivery_ladder.py (15) cover confirmed/
unverifiable/suspected_noop/degraded/old-driver verdicts, delivery_mode gating +
foreground_unsupported, and session-scoped foreground approval. Existing 265
computer_use tests still green.
Live E2E (real cua-driver 0.8.3 + tldraw offline on Linux/X11): a background click
returned effect='unverifiable'/path='ax' (no fabricated success), and a foreground
request returned code='foreground_unsupported' — correct on a driver that predates
the input.delivery_mode capability.
Adds two tests for the _background_review_write_guard manual-skill check:
- refuses delete on a skill with created_by=None (manually authored)
- allows delete on a skill with created_by='agent' (agent-created)
Use the direct POSIX parent relationship instead of process creation time and pid_exists checks. Remove the dead create-time argument chain while preserving process-group cleanup and signal forwarding.\n\nRefs #62505
A leaf subagent is meant to be denied delegate_task, execute_code, memory,
clarify, cronjob, and send_message. _strip_blocked_tools() only drops a
toolset when EVERY tool in it is blocked, so mixed platform bundles
(hermes-cli, hermes-telegram, and every other gateway bundle) survived
stripping and re-exposed the blocked tools after composite expansion. A
leaf child spawned from any gateway platform could recursively delegate,
run code, and write memory.
Pass exact one-tool deny toolsets into the child's disabled_toolsets so
model_tools subtracts the blocked names AFTER composite expansion, and the
restriction survives later registry/MCP refreshes. Orchestrators regain
only delegate_task.
Salvaged from #66036 by Mason Tanguay (@DictatorBacon); scoped to the
authority fix + its regressions (docs/interrupt changes dropped).
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
A root-launched CLI session can leak /root into the terminal cwd state a
non-root gateway/cron process later resolves (#65583). os.path.isdir('/root')
is True for a non-root user — stat only needs search permission on / — so
_resolve_safe_cwd returned it and subprocess.Popen(cwd='/root') died with
PermissionError: [Errno 13], failing EVERY cron job's terminal/file/search
tool on every command until restart.
_resolve_safe_cwd now requires X_OK (new _cwd_usable helper) and climbs to
the nearest enterable ancestor, logging a WARNING that names the leak class
when an existing-but-denied cwd is skipped. Missing-cwd recovery (#17558)
behavior unchanged.
E2E-verified: LocalEnvironment constructed with an unenterable cwd now runs
commands from the fallback directory instead of raising.
test_build_child_agent_ignores_acp_command_when_binary_missing patches
shutil.which globally and asserted the LAST call was which("copilot").
That is order-dependent: an unrelated which("uv") reached later in the same
process (which happens under some CI test-slice orderings) becomes the last
call, so assert_called_with("copilot") fails even though the copilot binary
was probed exactly as intended. Switch to assert_any_call("copilot"), which
verifies the actual intent and is robust to unrelated which() calls. The
behavioural assertions (provider, acp_command, acp_args) are unchanged.
Port from anomalyco/opencode#35439/#35500: preserve full MCP catalogs
across paginated tools/list responses.
The MCP spec allows servers to paginate tools/list, resources/list, and
prompts/list via an opaque nextCursor token. The Python SDK's
ClientSession.list_* methods fetch exactly one page per call, and hermes
never passed the cursor back — on a paginated server every tool,
resource, and prompt past page 1 was silently invisible to the agent.
Adds _paginate_full_list() (cursor-draining helper with a 50-page
runaway cap and spec-correct opaque-string cursor validation) and
applies it at all three discovery sites: _discover_tools(), the
tools/list_changed refresh handler, and the list_resources/list_prompts
utility handlers. The keepalive probe intentionally keeps its
single-page call (liveness only).
E2E: real stdio MCP server serving 3 pages (2/1/1 tools) — old code
discovered 2 tools, new code discovers all 4.
Port from openai/codex#31494: user-visible history replay must strip CSI
sequences and control characters. Stored conversation history can carry
raw terminal escapes (pasted content, gateway-origin text, model output
echoing injected tool results). Replaying it via /resume's recap panel or
build_recap (/status on CLI + gateway) wrote those bytes straight to the
terminal — an injected message could clear the screen, retitle the window,
move the cursor, or restyle the recap UI. Rich's Text() does not neutralize
raw escape bytes.
- tools/ansi_strip.py: add sanitize_display_text() — strip_ansi() plus
bare C0/C1 control removal, preserving \n and \t, normalizing \r to \n
(adapted to Python from Codex's sanitize_user_text; reuses the existing
ECMA-48 stripper instead of transcribing their char-walk)
- hermes_cli/cli_agent_setup_mixin.py: sanitize user + assistant text in
_display_resumed_history() before building the Rich recap panel
- hermes_cli/session_recap.py: sanitize preview lines in build_recap()
(_truncate choke point) so /status recaps are clean on every platform
- tests: 10 new sanitize_display_text cases (incl. the exact codex#31494
fixture), recap + resume-display leak assertions
