Replace the runner's tests/docker auto-cap heuristic with explicit
width control:
- run_tests.sh forwards HERMES_TEST_WORKERS through its hermetic env -i
(previously silently stripped — the documented override never worked
through the wrapper)
- run_tests_parallel.py drops the _DOCKERD_BOUND_JOBS special-case; the
suite-specific knowledge moves to the one place that runs that suite
- docker.yml pins HERMES_TEST_WORKERS=8. Width sweep with prewarmed
image + split files: -j4 58-62s, -j8 39s, -j12 35s w/ ~2x per-file
contention inflation; 8 is the knee.
Chain verified end-to-end: env var reaches the runner (6-worker probe),
no cap message on docker-only file lists, full suite 53/53 in 37.8s
at -j8.
The docker suite's wall time was max(whale files): four files each
serialized 2-3 ~110s container boots internally while 21 fast files
finished in seconds (P50 9.9s vs max 341s on the ARC runners). The
per-file parallel runner can only overlap what lives in separate files.
- test_dashboard.py -> 3 files (one boot each); shared _http_probe
helper moves to conftest
- test_container_restart.py -> 2 files (restart_container fixture
travels via the shared header; per-file container isolation is the
point of the split)
- test_docker_exec_privilege_drop.py -> boot-heavy e2e login test split
out; the two fast tests stay together
- test_config_migration.py: single test, unchanged
53 tests before and after, zero assertions changed — pure file
reorganization. Local (-j4, same cap as CI): 374.6s -> 58.0s wall,
slowest file 341.7s -> 18.6s.
docker.yml now requests id-token: write for WIF cache auth, but it is
invoked as a reusable workflow from ci.yml on PRs — and a called
workflow cannot request a permission its caller lacks. That mismatch
is a startup_failure (run 30596417013 died before any job). Add
id-token: write to ci.yml's top-level permissions.
Swap type=gha buildx cache for type=registry against
us-central1-docker.pkg.dev/.../ci-cache — same region as the ARC
runners, so layer blobs stop round-tripping to GitHub's cache CDN on
every build/rerun.
Auth is keyless both ways: reads ride the runner pod's GKE Workload
Identity (no login needed for cache-from); writes exchange the
workflow's GitHub OIDC token via WIF (google-github-actions/auth,
fork-guarded — fork PRs build cache-cold exactly like type=gha).
publish keeps its own unconditional WIF auth (trusted contexts only).
Infra: hermes-agent-ci-infra ef8dfb2.
1. test_tui_gateway_server: the two _start_agent_build tests waited only
for the _make_agent 'built' event, then popped the session while
_build's tail was still running. The tail's session.info/error emit
then landed on whatever _real_stdout a LATER test had patched in —
the write_json concurrency test intermittently saw 9 lines instead
of its own 8 (2-in-5 repro locally). Join the build thread (exposed
as session['_agent_build_thread']) before popping. 10/10 clean
full-file runs post-fix, was 2/5 failing.
2. docker.yml: the profile-action conversion dropped the step env —
including HERMES_TEST_IMAGE, so all 25 per-file subprocesses each
docker-built the image inside dind concurrently. That is the root
cause of the 15-minute docker jobs and the teardown timeout storms.
Restore it plus the blank-API-key policy vars.
Root cause of the last two amd64 docker failures: attempt 1 of a flaky
file times out mid-teardown (busy dind), the stale hermes-test-* name
survives, and the file-retry's docker run fails with a name Conflict —
so the retry mechanism itself was poisoned. The fixture now removes the
name BEFORE the test (fresh subprocess retry gets a clean slate), and
teardown swallows a slow-daemon TimeoutExpired instead of erroring a
passing test (1 passed, 1 error -> 1 passed).
Move the dockerd-bound worker cap from a workflow env var into
run_tests_parallel.py: when every file in the run is under
tests/docker/, cap -j at 4 (the suite shares one docker daemon; width
beyond that thrashes it — files stretch ~100s -> ~900s and teardown
docker-rm calls blow their 10s timeout). Explicit -j or
HERMES_TEST_WORKERS always wins; mixed file lists are unaffected.
Verified: docker-only list caps 32->4, -j 12 respected, mixed list
uncapped. Drops the HERMES_TEST_WORKERS=4 pin from docker.yml.
Diagnosis from the profiler + runner logs: with the cgroup-aware
default (-j 16 on the 8-CPU pods) all 25 files run concurrently against
the single dind daemon. Every file stretches to ~900s wall (P50 892s,
CPU-wall 14549s vs 918s wall) and teardown docker-rm calls exceed their
10s timeout — the job dies on teardown errors while tests themselves
pass 53/53. The arm64 lane (2 CPU → -j 4) went green for exactly this
reason. Pin the amd64 lane to the same effective width.
- doctor: 'inside a container' branch rewrote terminal_env to local for
EVERY non-docker backend, so TERMINAL_ENV=vercel_sandbox diagnostics
vanished when doctor ran inside a container (CI runner pods). Scope
the informational skip to the implicit local case only; remote
backends keep their real diagnostics. Fixes
test_doctor_reports_vercel_backend_diagnostics on ARC runners — and
for actual users running doctor in the Docker distribution with a
remote terminal backend configured.
- termux audio test: detect_audio_environment() probes the real host
for containment; pin is_container=False (a Termux device is never a
container) so the containerized runner doesn't flip available=False.
os.cpu_count() reports the HOST cores. In an ARC runner pod
(limit 8 CPU on a 22-core node) the runner spawned -j 44 workers on
8 usable CPUs — ~5x oversubscription. Every 'timing flake' family on
the self-hosted runners (docker rm teardown TimeoutExpired x111,
compression fork, termux probe, pty reaper, session hygiene) is CPU
starvation from that oversubscription, not real test bugs.
Read cgroup v2 cpu.max (v1 cfs_quota fallback) and clamp to host count.
Verified: --cpus=8 container reports 8, bare host unchanged.
HERMES_TEST_WORKERS override still wins.
- test_supports_systemd_services_returns_true_when_systemctl_present:
pin is_container=False (host contract; CI runner pods are containers)
- test_systemd_restart_gracefully_restarts_running_service_and_waits:
stub _preflight_user_systemd — no user D-Bus in runner pods; the test
asserts restart choreography, not D-Bus reachability
Test fixes for ARC runner pods (containers) vs GHA ubuntu-latest VMs:
- test_gateway_wsl / test_copilot_acp_client: pin is_container=False on
host-behavior tests — runner pods ARE containers and the prod code
intentionally behaves differently there. Also drop inherited
HERMES_REAL_HOME so nix dev shells don't leak into the assert.
- honcho memo + skill-utils external-dirs cache: add st_size to the
mtime_ns cache keys. overlayfs (runner pods) coalesces rapid writes
into one mtime tick, so same-tick edits were served stale (3 honcho
pin tests + skill cache invalidation test).
Verified with KUBERNETES_SERVICE_HOST set to simulate the pod env.
Swap all `runs-on: ubuntu-latest` to `runs-on: arc-runner-set` all jobs.
The ARM docker build job in docker.yml uses `${{ matrix.runner }}`
and is left untouched since the GKE runner pool is x86_64 only.
Runners are backed by ARC (Actions Runner Controller) on a GKE cluster
with a spot preemptible node pool that scales based on job demand.
Use the baked Electron dependencies for the desktop E2E job.
The repo's .npmrc sets engine-strict=true and package.json pins
engines.npm, so an npm outside that range aborts every npm ci /
npm install we run inside the checkout:
npm error code EBADENGINE
npm error notsup Required: {"npm":"<11.10.0 || >=12.0.0"}
npm error notsup Actual: {"npm":"11.10.0"}
Our callers made that worse: _run_npm_install_deterministic sees
`npm ci` fail and falls through to `npm install`, which fails
identically, so the user got a buried EBADENGINE and no remedy.
React to the failure instead of predicting it. npm states the
required range in its own error, so there is no need for a version
probe on the happy path or a semver range matcher — the recovery
reads the constraint out of the output it just produced, upgrades,
and retries once.
Scope is deliberately narrow. Hermes only upgrades an npm inside its
own managed Node tree ($HERMES_HOME/node), installing with --prefix
so bin/npm keeps resolving to the upgraded lib/node_modules/npm; a
managed install writes prefix=~/.local into node/etc/npmrc, so
without the override the "upgrade" would land elsewhere while the
managed npm stayed stale. A system / nvm / brew / Nix npm belongs to
the user, so that case prints the exact command and lets the original
failure stand.
The upgrade runs from a temp cwd with npm_config_min_release_age=0,
otherwise the checkout's own min-release-age gate would refuse the
npm release we need.
_run_npm_install_deterministic's capture_output=False callers (the
desktop install) streamed npm output and returned stderr=None, which
would leave the recovery nothing to read — stderr is now teed, so
live output is unchanged and the text stays inspectable.
Verified end to end against real npm binaries on copies of a managed
tree: managed npm 11.10.0 -> EBADENGINE -> upgraded to 12.0.2 ->
retry exits 0; a foreign npm 11.10.0 hard-fails with the manual
command and is left untouched.
pin brace-expansion to 5.0.8
update concurrently to 10.0.4
update electron-builder to 26.15.3
update eslint to 10.8.0
update eslint-plugin-perfectionist to 5.10.0
update @assistant-ui/react to 0.15.0
update @assistant-ui/react-streamdown to 0.3.8
update radix-ui to 1.6.7
update react-router-dom to react-router@8.3.0 - react-router-dom is no longer a standalone package, it just reexports react-router
remove @radix-ui/react-slot: we import this from `radix-ui`
remove eslint-plugin-react: we imported it, but never actually used it!
The statusbar is now opt-in. Existing users with a stored preference
keep their choice; new users get a clean bottom edge. The way back is
the view.toggleStatusbar keybind or the ⌘K row, unchanged.
Dated snapshot of deepseek-v4-flash, live on both provider endpoints
(verified via OpenRouter /api/v1/models and Nous portal /v1/models).
Context (1M via deepseek-v4-flash prefix match), reasoning stale-timeout
floor (600s), and pricing (official_models_api live billing on both
routes) all resolve without new entries. model-catalog.json regenerated
via scripts/build_model_catalog.py.
