hermes-agent-ci-infra now offers three amd64 scale sets instead of one,
split by capability rather than size:
arc-runner-set no dind 4/8Gi general (unchanged label)
arc-runner-docker dind 4/8Gi needs a docker daemon
arc-runner-small no dind 500m/1Gi short gates, warm pod
Of the 22 workflows targeting arc-runner-set, only docker.yml ever talks to
a daemon. Every other job was booting a privileged docker:dind sidecar, an
extra init container and a `docker info` startup probe just to run ruff, so
dind now lives only on arc-runner-docker (and arm64, which already had it).
docker.yml's amd64 legs and its manifest-merge job move there; the arm64
legs are unchanged.
The short gate jobs move to arc-runner-small. Each of these ran for 14-29s
while reserving 4 CPU / 8Gi — a fifth of a c3-standard-22 node — and there
are enough of them per PR to delay the test matrix they run alongside. The
small set sits on the always-on system pool and keeps one warm pod, so
these jobs skip pod creation and image pull entirely.
No behavior change to what any job does; only where it runs.
Eleven jobs on every push repeated the same three network round-trips
before doing any work: download ripgrep from GitHub releases, run
astral-sh/setup-uv, then `uv python install 3.11`. The 8 test slices,
e2e, lint x2, docker tests, and uv-lockfile-check all paid it, all for
identical bytes. Each hop was also a failure mode — the 2026-07-28
slice-5 incident was a transient setup-uv manifest fetch failing a whole
job, and pinning the version narrowed that window without closing it.
hermes-agent-ci-infra now bakes ripgrep 15.1.0, uv 0.9.28, and CPython
3.11 into nousresearch/nous-gke-runner (same versions, so this is a move
not an upgrade), so these steps are pure overhead. Remove them.
The wheel cache is the one part of setup-uv still worth having: it is
per-workspace, not per-image, and without it `uv sync` re-downloads and
re-builds every wheel — the toolchain would be faster to set up and the
sync dramatically slower, a net loss. Replace `enable-cache: true` with
a small .github/actions/uv-cache composite doing the same actions/cache
on ~/.cache/uv, keyed on pyproject.toml + uv.lock. runner.arch is in the
key because the cache holds built wheels and docker.yml runs on arm64
too; the restore-keys prefix means a stale hit still saves most of the
download, and `uv sync --locked` re-resolves from uv.lock regardless so
a partial hit cannot produce a wrong environment.
lint.yml and uv-lockfile-check.yml only `uv tool install` / `uv lock
--check` and never build a project venv, so they drop the setup step
without needing the cache action at all.
Verified against the built image, running as the `runner` user with
`--network none` so nothing can silently re-download: rg 15.1.0, uv
0.9.28, and `uv python find 3.11` all resolve. With hermes-agent's real
pyproject.toml and uv.lock and no setup step of any kind, `uv sync
--locked --python 3.11 --extra dev` completes in 3s into a working
3.11.14 venv. actionlint is clean (the remaining arc-runner-set and
SC2016 warnings are pre-existing on main).
Depends on the image change landing first: pods pull :latest on start,
so merging this before the image is pushed breaks every runner.
The PR-read-only hardening (1d5eb3bf4) gated the WIF auth step on
non-PR events, which also silently gated the Artifact Registry docker
login that step fed. The comment said "PRs read the cache via the pod's
GKE Workload Identity", but pod WI is not ambient for buildx: with no
login, cache-from does an anonymous pull against us-central1-docker.pkg.dev
and gets 403 Forbidden (visible as "failed to configure registry cache
importer" in every PR build since), so every PR built cache-cold
(~12-16 min instead of ~2-3).
Fix: on PR events, mint an access token from the pod's GKE metadata
server (the runner pod's KSA impersonates gha-buildx-cache-ro@, which
has only artifactregistry.reader) and feed it to the same docker login
step. The security boundary is unchanged — PR builds still cannot write
cache layers; cache-to remains gated on the WIF token that only exists
on trusted main-push/release contexts.
Address review findings on the ARC migration:
- docker.yml: WIF auth (and therefore Artifact Registry cache WRITES)
now only runs on non-PR events. The build job runs PR-controlled code
and the publish job reads the same buildcache ref, so a PR-writable
cache was a layer-poisoning vector. PRs of any origin keep cache
READS via the runner pod's GKE Workload Identity — that's where the
15min -> 2-3min win comes from; main pushes repopulate writes.
- docker.yml: profile label is now docker-tests-<arch>. Both matrix
legs uploaded resource-profile-docker-tests; upload-artifact v4+
rejects the duplicate and continue-on-error swallowed it, silently
dropping one arch's profile.
- actions/profile: run the wrapped command with bash -eo pipefail to
match normal `run:` step semantics (a failing `source .venv/...`
must fail the step, not fall through).
- js/e2e/site workflows: bake node22 into the node_modules cache keys
so a future node-version bump can't restore stale native builds
(node-pty, electron postinstall) against an unchanged lockfile.
- test_container_restart_stale_pid: forward deadline_s/interval_s to
wait_for_log instead of silently dropping them.
- doctor.py: refresh a stale comment on the in-container docker branch.
Replace the runner's tests/docker auto-cap heuristic with explicit
width control:
- run_tests.sh forwards HERMES_TEST_WORKERS through its hermetic env -i
(previously silently stripped — the documented override never worked
through the wrapper)
- run_tests_parallel.py drops the _DOCKERD_BOUND_JOBS special-case; the
suite-specific knowledge moves to the one place that runs that suite
- docker.yml pins HERMES_TEST_WORKERS=8. Width sweep with prewarmed
image + split files: -j4 58-62s, -j8 39s, -j12 35s w/ ~2x per-file
contention inflation; 8 is the knee.
Chain verified end-to-end: env var reaches the runner (6-worker probe),
no cap message on docker-only file lists, full suite 53/53 in 37.8s
at -j8.
Swap type=gha buildx cache for type=registry against
us-central1-docker.pkg.dev/.../ci-cache — same region as the ARC
runners, so layer blobs stop round-tripping to GitHub's cache CDN on
every build/rerun.
Auth is keyless both ways: reads ride the runner pod's GKE Workload
Identity (no login needed for cache-from); writes exchange the
workflow's GitHub OIDC token via WIF (google-github-actions/auth,
fork-guarded — fork PRs build cache-cold exactly like type=gha).
publish keeps its own unconditional WIF auth (trusted contexts only).
Infra: hermes-agent-ci-infra ef8dfb2.
1. test_tui_gateway_server: the two _start_agent_build tests waited only
for the _make_agent 'built' event, then popped the session while
_build's tail was still running. The tail's session.info/error emit
then landed on whatever _real_stdout a LATER test had patched in —
the write_json concurrency test intermittently saw 9 lines instead
of its own 8 (2-in-5 repro locally). Join the build thread (exposed
as session['_agent_build_thread']) before popping. 10/10 clean
full-file runs post-fix, was 2/5 failing.
2. docker.yml: the profile-action conversion dropped the step env —
including HERMES_TEST_IMAGE, so all 25 per-file subprocesses each
docker-built the image inside dind concurrently. That is the root
cause of the 15-minute docker jobs and the teardown timeout storms.
Restore it plus the blank-API-key policy vars.
Move the dockerd-bound worker cap from a workflow env var into
run_tests_parallel.py: when every file in the run is under
tests/docker/, cap -j at 4 (the suite shares one docker daemon; width
beyond that thrashes it — files stretch ~100s -> ~900s and teardown
docker-rm calls blow their 10s timeout). Explicit -j or
HERMES_TEST_WORKERS always wins; mixed file lists are unaffected.
Verified: docker-only list caps 32->4, -j 12 respected, mixed list
uncapped. Drops the HERMES_TEST_WORKERS=4 pin from docker.yml.
Diagnosis from the profiler + runner logs: with the cgroup-aware
default (-j 16 on the 8-CPU pods) all 25 files run concurrently against
the single dind daemon. Every file stretches to ~900s wall (P50 892s,
CPU-wall 14549s vs 918s wall) and teardown docker-rm calls exceed their
10s timeout — the job dies on teardown errors while tests themselves
pass 53/53. The arm64 lane (2 CPU → -j 4) went green for exactly this
reason. Pin the amd64 lane to the same effective width.
Swap all `runs-on: ubuntu-latest` to `runs-on: arc-runner-set` all jobs.
The ARM docker build job in docker.yml uses `${{ matrix.runner }}`
and is left untouched since the GKE runner pool is x86_64 only.
Runners are backed by ARC (Actions Runner Controller) on a GKE cluster
with a spot preemptible node pool that scales based on job demand.
Use the baked Electron dependencies for the desktop E2E job.
The Docker Build, Test, and Publish workflow fails when
docker/setup-buildx-action can't pull the moby/buildkit:buildx-stable-1
image from Docker Hub. The failure happens during builder bootstrap at
the auth token exchange — a transient network blip (connection reset,
read timeout, rate limiting) that self-resolves on re-run.
Recent failure (run 30449230291, merge job):
read tcp 10.1.0.171:45666->104.18.43.178:443: read: connection reset by peer
This has hit us before and will again — it's the same class of
transient Docker Hub flake that the merge job already retries for
imagetools create. But buildx setup had no retry, so a single network
hiccup killed the entire job (build, publish, or merge) even though
nothing was wrong with the code or the image.
Fix: wrap each of the 3 buildx setup steps (build, publish, merge jobs)
with continue-on-error + a conditional retry step. The maintained action
is preserved as-is — we just give it a second attempt if the first
fails. The action generates a unique builder name per invocation, so the
retry never collides with the failed first attempt. The second attempt
has no continue-on-error, so genuine persistent failures still fail the
job.
The docker/setup-buildx-action maintainer has explicitly said retry
belongs at the workflow level, not inside the action [1], and other
repos use this same continue-on-error pattern for this exact issue [2].
[1] docker/setup-buildx-action#510
[2] joshjhall/containers#688, ethpandaops/eth-client-docker-image-builder#391
Unpinned, astral-sh/setup-uv resolves 'latest' by fetching
https://raw.githubusercontent.com/astral-sh/versions/.../uv.ndjson on
EVERY job. A transient failure of that fetch fails the whole job before
any test runs (2026-07-28: tests slice 5/8 died 12s in with
'##[error]fetch failed' on PR #73514). Pinning version makes setup-uv
download the binary directly — one less external hop per job across all
7 call sites (tests, lint x2, docker, e2e-desktop, lockfile-check).
* fix(ci): publish inline E2E evidence
Upload bounded screenshot evidence from E2E, then publish validated images
from a trusted workflow_run job to commit-pinned branches in the evidence repo.
Wait briefly for the live CI review comment marker before publishing, so
GitHub's read-after-write delay cannot leave an orphaned evidence branch.
* fix(ci): isolate privileged credentials from PR jobs
Keep App private keys and Docker Hub credentials out of PR-controlled
workflows. Use protected environments for trusted publishing and a public
repository variable for the App client ID.
* fix(ci): attach E2E evidence with restricted bot session
Replace the App-backed evidence repository publisher with gh-image uploads
from a dedicated bot session in the gh-image environment.
* fix(ci): publish validated E2E evidence from forks
Let the trusted default-branch publisher handle bounded, validated evidence
artifacts from fork PR CI without checking out or executing fork code.
* feat(attribution): conflict-free contributor mappings via contributors/emails/ directory
The AUTHOR_MAP dict in scripts/release.py was a merge-conflict magnet:
every concurrent salvage PR appended entries to the same lines of the
same file, so parallel PRs re-conflicted on every merge to main.
New system: one file per email under contributors/emails/ — filename is
the commit-author email, first non-comment line is the GitHub login.
File additions never conflict, so any number of PRs can add mappings
concurrently.
- scripts/release.py: AUTHOR_MAP is now LEGACY_AUTHOR_MAP (frozen)
merged with the directory at import time (directory wins). All
existing consumers (resolve_author, contributor_audit.py) unchanged.
- scripts/add_contributor.py: idempotent CLI to add a mapping; refuses
conflicting reassignments (incl. against the legacy map), validates
email/login shapes.
- contributor-check.yml: attribution gate now accepts a mapping file OR
a legacy entry; failure message prints the exact add_contributor
command. Also auto-resolves bare <login>@users.noreply.github.com
emails is intentionally NOT added (kept id+login form only, matching
previous behavior).
- contributor_audit.py: guidance now points at add_contributor.py.
- tests/scripts/test_contributor_map.py: 12 tests covering loader,
merge precedence, CLI idempotency/conflict/validation, subprocess E2E.
* feat(ci): one-shot per-file flake retry in the parallel test runner
A failing test FILE is re-run once in a fresh subprocess. Pass-on-retry
counts as green but is loudly reported in a '⚠ FLAKY' summary section
(with both attempts' output preserved) so the flake gets fixed instead
of eating a full-run rerun. Deterministic failures fail both attempts —
regressions cannot be laundered green.
- --file-retries N / HERMES_TEST_FILE_RETRIES (default 1, 0 disables)
- E2E verified: simulated first-run-fail flake goes green with banner;
deterministic failure still exits 1; retries=0 restores old behavior.
This converts the dominant CI failure mode (one timing-sensitive test
flaking a 4600-test shard, requiring a manual 10-minute rerun and an
agent triage loop) into a self-healing retry that costs one file's
runtime.
* test(approval): loosen wall-clock perf bounds 0.15s -> 2.0s
These guard against catastrophic regex backtracking (seconds-to-minutes
class), but 0.15s is within scheduler-stall noise on loaded shared CI
runners — test_max_accepted_separator_free_input_is_fast failed a CI
shard this week on runner load alone. 2.0s still catches the regression
class with zero flake surface.
* fix(ci): job timeouts everywhere + retries on all network installs
Reliability pass over every workflow:
- timeout-minutes on all 21 jobs that lacked one (a hung job previously
burned the 6-hour default runner budget)
- ./.github/actions/retry wrapped around every network-fetching install
that lacked it: pip installs (deploy-site, skills-index), npm ci
(deploy-site website, upload_to_pypi web + ui-tui), uv sync (docker
test deps). Deterministic build steps (npm run build) deliberately
NOT retried — split into separate steps so a real build failure fails
fast instead of retrying 3x.
* docs(agents): document the file-retry flake policy
* fix(ci): curl retries on deploy hook + skills-index probe
* fix(ci): kill the remaining transient-failure classes in workflows + Dockerfile
From the workflow reliability audit:
- tests.yml: duration-cache restore had NO restore-keys while saves use
run_id-suffixed keys — the cache never matched once, so LPT slicing
always ran blind and unbalanced slices pushed heavy files toward the
per-file timeout. One-line restore-keys fixes slice balancing.
- Label gates (lint ci-reviewed, supply-chain mcp-catalog-reviewed):
'gh pr view || true' turned an API blip into 'label absent' → false
BLOCKING failure. Now 3x retry, and API failure is reported as an API
failure instead of a missing label.
- detect-changes action: compare API retried before failing open (was
silently running all lanes on any blip).
- uv-lockfile-check: 'uv lock --check' resolves against PyPI — retried
so registry blips don't read as 'lockfile stale'.
- docker.yml merge job: imagetools create retried (Docker Hub eventual
consistency on just-pushed digests).
- Dockerfile: apt-get Acquire::Retries=3; s6-overlay ADDs converted to
curl --retry 3 (ADD cannot retry; checksums still enforced); npm
--fetch-retries=5; playwright chromium fetch retried 3x.
- Advisory artifact uploads (per-slice durations, ci-timings report)
get continue-on-error so an artifact-service blip can't fail a green
test slice.
* fix(tests): kill the two root-cause flakes — leaking pre-warm timer + env-dependent provider list
- test_tui_gateway_server.py: session.create / non-eager session.resume
arm a 50ms threading.Timer (_schedule_agent_build) that outlives its
test and fires into the NEXT test's _make_agent mock, racily
corrupting captured state (the recurring session_resume shard
failures). Replaced the per-test whack-a-mole stub with a module-wide
autouse fixture; the 3 worker-lifecycle tests that genuinely need the
deferred build opt back in via @pytest.mark.real_agent_prewarm (new
marker in pyproject).
- test_api_key_providers.py: PROVIDER_ENV_VARS is now derived from the
live PROVIDER_REGISTRY instead of a hand-list that had drifted
(missing HF_TOKEN / DEEPINFRA_API_KEY) — resolve_provider('auto')
tests failed on any machine with HF_TOKEN exported. E2E-verified with
HF_TOKEN/DEEPINFRA_API_KEY set: 42/42 pass.
* test: de-flake 30 timing-sensitive test files for loaded CI runners
Root-cause fixes from the flake audit (session-DB mining + repo sweep):
Event-based sync instead of sleep-sync:
- title_generator: mock sets threading.Event, wait(10) replaces
sleep(0.3) hoping the daemon thread got scheduled
- docker zombie_reaping / profile_gateway: poll-for-state helpers
replace fixed 1-3s sleeps (s6 transitions + SIGCHLD reaping are async)
- process_registry tree test: select()-bounded readline replaces an
unbounded blocking read (parent wedge now fails THIS test with a clear
message instead of an opaque rc=124 file kill); SIGTERM grace 1s->2s
(the 1s partition window mid-interpreter-startup is how a child PID
escaped the live-system guard in CI)
Timeout raises (loaded 8-way-sliced runners see ~5s scheduling floors;
all of these complete in ms-to-1s when healthy so the raises cost
nothing on green runs):
- subprocess/thread waits <= 2s raised to 10-15s across mcp_tool,
mcp_circuit_breaker, mcp_reconnect_retry_reset, mcp_parked_self_probe,
mcp_cancelled_error_propagation, registry, clarify_gateway, interrupt,
voice_cli_integration, docker_environment, session_store_lock_io,
planned_stop_watcher, cli_interrupt_subagent, thread_scoped_output
(joins now also assert not is_alive() so stragglers fail loudly)
- wall-clock discrimination ceilings loosened where the guarded hang is
10x larger: local_background_child_hang 4s->10s, interrupt_cleanup
setup 5s->20s + pgid-exit 30s->60s, mcp_stability grandchild spinup
5s->15s, protocol/gil-starvation fast-handler 0.5s->2s,
iso_certify_seam 1.5s->5s, wait_for_mcp_discovery 0.1s->1s
- narrow assertion windows widened: honcho first-turn wait 0.4..0.65 ->
0.25..2.0 (property is bounded-not-hung, not an exact wall-clock);
compression fork-lock TTL 1s->3s (12 refresh chances per lease);
compression-lock expiry margins symmetric (ttl 0.05->0.5, sleep 1.0)
- telegram hung-DNS bound 1.0->1.4 (fake hang is 1.5s — must stay under)
* fix(tests): repair indentation from de-flake batch edit
* fix(tests): harden env isolation and replace remaining sleep-sync races
The full 42k-test run and complete npm check surfaced three more classes:
- Environment isolation: local ~/.honcho defaultHost and SSH_* variables
leaked into Python/TUI tests. Pin the default Honcho host in the
hermetic fixture, isolate the one fallback test from ~/.honcho, and
blank SSH_* around terminalSetup tests. This flipped 20 false failures
back to deterministic behavior on developer machines.
- Background-thread sleep-sync: Honcho async writer tests patched
time.sleep globally, then busy-polled with that same mocked sleep. Under
full-suite load the poller could starve the writer. Each test now waits
on an Event emitted by the exact flush/retry transition; 30/30 passed
under 15-way contention.
- Desktop streaming: the test slept 80ms and assumed a 500ms timer could
not fire before its assertion. A loaded runner descheduled the test for
>500ms and both chunks arrived. Producer controls now gate second-chunk
and completion transitions explicitly.
Also make file-retry observability complete: a self-healed flaky file now
prints BOTH attempts' full output in the FLAKY summary. Two behavioral
runner tests prove pass-on-retry is green+loud+traceback-preserving, while
a deterministic failure remains red.
* refactor(ci): use gh bot pat, better retries
refactor(ci): use retry action for PR label fetch
the retry action now captures stdout as a step output, so it can serve
double duty: retry + output capture for commands like 'gh pr view' whose
result must be consumed by later steps.
Retry action gains:
- 'stdout' output (heredoc-delimited to preserve newlines)
- tee to temp file so stdout still streams to the job log
- step id 'retry' for output reference
Both lint.yml and supply-chain-audit.yml now use the retry action
directly with 'command: gh pr view ...' and read
steps.<id>.outputs.stdout.
ci: use AUTOFIX_BOT_PAT for all gh CLI / GitHub API auth
Replace secrets.GITHUB_TOKEN and github.token with
secrets.AUTOFIX_BOT_PAT across all workflows and composite actions
that use the gh CLI or GitHub API. The PAT has consistent permissions
across fork PRs (where GITHUB_TOKEN is read-only), avoids API rate
limit sharing with the default token, and is already used by
js-autofix.yml for the same reasons.
19 sites swapped across 9 files:
- lint.yml (3): label fetch, comment post/edit, comment update
- supply-chain-audit.yml (5): scan, critical comment, unbounded dep
comment, label fetch, mcp-catalog comment
- lockfile-diff.yml (1): PR comment post/update
- skills-index-freshness.yml (1): issue creation on degraded probe
- skills-index.yml (2): index build, trigger deploy workflow
- upload_to_pypi.yml (2): release view poll, release upload
- ci.yml (1): timings report
- deploy-site.yml (2): skills index crawl
- detect-changes/action.yml (1): compare API call
---------
Co-authored-by: ethernet <arilotter@gmail.com>
lint.yml inlined github.head_ref (the fork PR branch name, attacker-
controlled) into the diff-summary run: block. GitHub expands ${{ }} into
the script text before bash tokenizes it, so a branch like x$(id) runs on
the lint runner. The pull_request trigger keeps the token read-only, but
the sink still allows CI resource abuse and cache/artifact tampering, and
would become RCE-with-secrets under pull_request_target.
Route head_ref through an env var (env values are not subject to expression
injection) and reference "$HEAD_REF". Apply the same to the two docker.yml
sites that interpolate github.event.release.tag_name.
Fixes GHSA-jpw6-c7jr-c56v, GHSA-2843-hjmf-7x96.
Credit: @technotion, @youngstar-eth.
