The PR-read-only hardening (1d5eb3bf4) gated the WIF auth step on
non-PR events, which also silently gated the Artifact Registry docker
login that step fed. The comment said "PRs read the cache via the pod's
GKE Workload Identity", but pod WI is not ambient for buildx: with no
login, cache-from does an anonymous pull against us-central1-docker.pkg.dev
and gets 403 Forbidden (visible as "failed to configure registry cache
importer" in every PR build since), so every PR built cache-cold
(~12-16 min instead of ~2-3).
Fix: on PR events, mint an access token from the pod's GKE metadata
server (the runner pod's KSA impersonates gha-buildx-cache-ro@, which
has only artifactregistry.reader) and feed it to the same docker login
step. The security boundary is unchanged — PR builds still cannot write
cache layers; cache-to remains gated on the WIF token that only exists
on trusted main-push/release contexts.
Address review findings on the ARC migration:
- docker.yml: WIF auth (and therefore Artifact Registry cache WRITES)
now only runs on non-PR events. The build job runs PR-controlled code
and the publish job reads the same buildcache ref, so a PR-writable
cache was a layer-poisoning vector. PRs of any origin keep cache
READS via the runner pod's GKE Workload Identity — that's where the
15min -> 2-3min win comes from; main pushes repopulate writes.
- docker.yml: profile label is now docker-tests-<arch>. Both matrix
legs uploaded resource-profile-docker-tests; upload-artifact v4+
rejects the duplicate and continue-on-error swallowed it, silently
dropping one arch's profile.
- actions/profile: run the wrapped command with bash -eo pipefail to
match normal `run:` step semantics (a failing `source .venv/...`
must fail the step, not fall through).
- js/e2e/site workflows: bake node22 into the node_modules cache keys
so a future node-version bump can't restore stale native builds
(node-pty, electron postinstall) against an unchanged lockfile.
- test_container_restart_stale_pid: forward deadline_s/interval_s to
wait_for_log instead of silently dropping them.
- doctor.py: refresh a stale comment on the in-container docker branch.
Replace the runner's tests/docker auto-cap heuristic with explicit
width control:
- run_tests.sh forwards HERMES_TEST_WORKERS through its hermetic env -i
(previously silently stripped — the documented override never worked
through the wrapper)
- run_tests_parallel.py drops the _DOCKERD_BOUND_JOBS special-case; the
suite-specific knowledge moves to the one place that runs that suite
- docker.yml pins HERMES_TEST_WORKERS=8. Width sweep with prewarmed
image + split files: -j4 58-62s, -j8 39s, -j12 35s w/ ~2x per-file
contention inflation; 8 is the knee.
Chain verified end-to-end: env var reaches the runner (6-worker probe),
no cap message on docker-only file lists, full suite 53/53 in 37.8s
at -j8.
The docker suite's wall time was max(whale files): four files each
serialized 2-3 ~110s container boots internally while 21 fast files
finished in seconds (P50 9.9s vs max 341s on the ARC runners). The
per-file parallel runner can only overlap what lives in separate files.
- test_dashboard.py -> 3 files (one boot each); shared _http_probe
helper moves to conftest
- test_container_restart.py -> 2 files (restart_container fixture
travels via the shared header; per-file container isolation is the
point of the split)
- test_docker_exec_privilege_drop.py -> boot-heavy e2e login test split
out; the two fast tests stay together
- test_config_migration.py: single test, unchanged
53 tests before and after, zero assertions changed — pure file
reorganization. Local (-j4, same cap as CI): 374.6s -> 58.0s wall,
slowest file 341.7s -> 18.6s.
docker.yml now requests id-token: write for WIF cache auth, but it is
invoked as a reusable workflow from ci.yml on PRs — and a called
workflow cannot request a permission its caller lacks. That mismatch
is a startup_failure (run 30596417013 died before any job). Add
id-token: write to ci.yml's top-level permissions.
Swap type=gha buildx cache for type=registry against
us-central1-docker.pkg.dev/.../ci-cache — same region as the ARC
runners, so layer blobs stop round-tripping to GitHub's cache CDN on
every build/rerun.
Auth is keyless both ways: reads ride the runner pod's GKE Workload
Identity (no login needed for cache-from); writes exchange the
workflow's GitHub OIDC token via WIF (google-github-actions/auth,
fork-guarded — fork PRs build cache-cold exactly like type=gha).
publish keeps its own unconditional WIF auth (trusted contexts only).
Infra: hermes-agent-ci-infra ef8dfb2.
1. test_tui_gateway_server: the two _start_agent_build tests waited only
for the _make_agent 'built' event, then popped the session while
_build's tail was still running. The tail's session.info/error emit
then landed on whatever _real_stdout a LATER test had patched in —
the write_json concurrency test intermittently saw 9 lines instead
of its own 8 (2-in-5 repro locally). Join the build thread (exposed
as session['_agent_build_thread']) before popping. 10/10 clean
full-file runs post-fix, was 2/5 failing.
2. docker.yml: the profile-action conversion dropped the step env —
including HERMES_TEST_IMAGE, so all 25 per-file subprocesses each
docker-built the image inside dind concurrently. That is the root
cause of the 15-minute docker jobs and the teardown timeout storms.
Restore it plus the blank-API-key policy vars.
Root cause of the last two amd64 docker failures: attempt 1 of a flaky
file times out mid-teardown (busy dind), the stale hermes-test-* name
survives, and the file-retry's docker run fails with a name Conflict —
so the retry mechanism itself was poisoned. The fixture now removes the
name BEFORE the test (fresh subprocess retry gets a clean slate), and
teardown swallows a slow-daemon TimeoutExpired instead of erroring a
passing test (1 passed, 1 error -> 1 passed).
Move the dockerd-bound worker cap from a workflow env var into
run_tests_parallel.py: when every file in the run is under
tests/docker/, cap -j at 4 (the suite shares one docker daemon; width
beyond that thrashes it — files stretch ~100s -> ~900s and teardown
docker-rm calls blow their 10s timeout). Explicit -j or
HERMES_TEST_WORKERS always wins; mixed file lists are unaffected.
Verified: docker-only list caps 32->4, -j 12 respected, mixed list
uncapped. Drops the HERMES_TEST_WORKERS=4 pin from docker.yml.
Diagnosis from the profiler + runner logs: with the cgroup-aware
default (-j 16 on the 8-CPU pods) all 25 files run concurrently against
the single dind daemon. Every file stretches to ~900s wall (P50 892s,
CPU-wall 14549s vs 918s wall) and teardown docker-rm calls exceed their
10s timeout — the job dies on teardown errors while tests themselves
pass 53/53. The arm64 lane (2 CPU → -j 4) went green for exactly this
reason. Pin the amd64 lane to the same effective width.
- doctor: 'inside a container' branch rewrote terminal_env to local for
EVERY non-docker backend, so TERMINAL_ENV=vercel_sandbox diagnostics
vanished when doctor ran inside a container (CI runner pods). Scope
the informational skip to the implicit local case only; remote
backends keep their real diagnostics. Fixes
test_doctor_reports_vercel_backend_diagnostics on ARC runners — and
for actual users running doctor in the Docker distribution with a
remote terminal backend configured.
- termux audio test: detect_audio_environment() probes the real host
for containment; pin is_container=False (a Termux device is never a
container) so the containerized runner doesn't flip available=False.
os.cpu_count() reports the HOST cores. In an ARC runner pod
(limit 8 CPU on a 22-core node) the runner spawned -j 44 workers on
8 usable CPUs — ~5x oversubscription. Every 'timing flake' family on
the self-hosted runners (docker rm teardown TimeoutExpired x111,
compression fork, termux probe, pty reaper, session hygiene) is CPU
starvation from that oversubscription, not real test bugs.
Read cgroup v2 cpu.max (v1 cfs_quota fallback) and clamp to host count.
Verified: --cpus=8 container reports 8, bare host unchanged.
HERMES_TEST_WORKERS override still wins.
- test_supports_systemd_services_returns_true_when_systemctl_present:
pin is_container=False (host contract; CI runner pods are containers)
- test_systemd_restart_gracefully_restarts_running_service_and_waits:
stub _preflight_user_systemd — no user D-Bus in runner pods; the test
asserts restart choreography, not D-Bus reachability
Test fixes for ARC runner pods (containers) vs GHA ubuntu-latest VMs:
- test_gateway_wsl / test_copilot_acp_client: pin is_container=False on
host-behavior tests — runner pods ARE containers and the prod code
intentionally behaves differently there. Also drop inherited
HERMES_REAL_HOME so nix dev shells don't leak into the assert.
- honcho memo + skill-utils external-dirs cache: add st_size to the
mtime_ns cache keys. overlayfs (runner pods) coalesces rapid writes
into one mtime tick, so same-tick edits were served stale (3 honcho
pin tests + skill cache invalidation test).
Verified with KUBERNETES_SERVICE_HOST set to simulate the pod env.
Swap all `runs-on: ubuntu-latest` to `runs-on: arc-runner-set` all jobs.
The ARM docker build job in docker.yml uses `${{ matrix.runner }}`
and is left untouched since the GKE runner pool is x86_64 only.
Runners are backed by ARC (Actions Runner Controller) on a GKE cluster
with a spot preemptible node pool that scales based on job demand.
Use the baked Electron dependencies for the desktop E2E job.
The repo's .npmrc sets engine-strict=true and package.json pins
engines.npm, so an npm outside that range aborts every npm ci /
npm install we run inside the checkout:
npm error code EBADENGINE
npm error notsup Required: {"npm":"<11.10.0 || >=12.0.0"}
npm error notsup Actual: {"npm":"11.10.0"}
Our callers made that worse: _run_npm_install_deterministic sees
`npm ci` fail and falls through to `npm install`, which fails
identically, so the user got a buried EBADENGINE and no remedy.
React to the failure instead of predicting it. npm states the
required range in its own error, so there is no need for a version
probe on the happy path or a semver range matcher — the recovery
reads the constraint out of the output it just produced, upgrades,
and retries once.
Scope is deliberately narrow. Hermes only upgrades an npm inside its
own managed Node tree ($HERMES_HOME/node), installing with --prefix
so bin/npm keeps resolving to the upgraded lib/node_modules/npm; a
managed install writes prefix=~/.local into node/etc/npmrc, so
without the override the "upgrade" would land elsewhere while the
managed npm stayed stale. A system / nvm / brew / Nix npm belongs to
the user, so that case prints the exact command and lets the original
failure stand.
The upgrade runs from a temp cwd with npm_config_min_release_age=0,
otherwise the checkout's own min-release-age gate would refuse the
npm release we need.
_run_npm_install_deterministic's capture_output=False callers (the
desktop install) streamed npm output and returned stderr=None, which
would leave the recovery nothing to read — stderr is now teed, so
live output is unchanged and the text stays inspectable.
Verified end to end against real npm binaries on copies of a managed
tree: managed npm 11.10.0 -> EBADENGINE -> upgraded to 12.0.2 ->
retry exits 0; a foreign npm 11.10.0 hard-fails with the manual
command and is left untouched.
pin brace-expansion to 5.0.8
update concurrently to 10.0.4
update electron-builder to 26.15.3
update eslint to 10.8.0
update eslint-plugin-perfectionist to 5.10.0
update @assistant-ui/react to 0.15.0
update @assistant-ui/react-streamdown to 0.3.8
update radix-ui to 1.6.7
update react-router-dom to react-router@8.3.0 - react-router-dom is no longer a standalone package, it just reexports react-router
remove @radix-ui/react-slot: we import this from `radix-ui`
remove eslint-plugin-react: we imported it, but never actually used it!
