hermes-agent-ci-infra now offers three amd64 scale sets instead of one,
split by capability rather than size:
arc-runner-set no dind 4/8Gi general (unchanged label)
arc-runner-docker dind 4/8Gi needs a docker daemon
arc-runner-small no dind 500m/1Gi short gates, warm pod
Of the 22 workflows targeting arc-runner-set, only docker.yml ever talks to
a daemon. Every other job was booting a privileged docker:dind sidecar, an
extra init container and a `docker info` startup probe just to run ruff, so
dind now lives only on arc-runner-docker (and arm64, which already had it).
docker.yml's amd64 legs and its manifest-merge job move there; the arm64
legs are unchanged.
The short gate jobs move to arc-runner-small. Each of these ran for 14-29s
while reserving 4 CPU / 8Gi — a fifth of a c3-standard-22 node — and there
are enough of them per PR to delay the test matrix they run alongside. The
small set sits on the always-on system pool and keeps one warm pod, so
these jobs skip pod creation and image pull entirely.
No behavior change to what any job does; only where it runs.
The "Generate slices" job spent ~90% of its wall time in actions/checkout
(4-27s across the last 10 runs, of a 10-32s job) pulling a ~212MB working
tree. Everything in the test matrix waits on it.
It doesn't need those files. `--generate-slices` returns before
`_approximately_count_tests`, so it only ever uses test file *paths* plus
the cached durations — it never opens a test file.
The blocker was discovery: `_discover_files` rglobs the filesystem, which
finds nothing under a sparse checkout. So add `--discover-from-git`, which
lists paths via `git ls-files`. Sparse checkout only clears the worktree
(entries are marked skip-worktree), so the index still carries every path
and enumerates exactly the same set. Skip-part filtering and the
root-override rule are duplicated to match `_discover_files` semantics.
Measured on a real clone of this repo:
full depth=1 clone: 7s 212M
blobless+sparse clone: 3s 3.4M (still sees all 2510 test paths)
Both discovery paths produce byte-identical slice JSON over the full 2472
test files, so slice assignment is unchanged.
Tests assert the properties that make this safe: the two discovery paths
agree on a full checkout, the git path still works when the files are
absent from disk, and the skip-part override behaves the same either way.
Eleven jobs on every push repeated the same three network round-trips
before doing any work: download ripgrep from GitHub releases, run
astral-sh/setup-uv, then `uv python install 3.11`. The 8 test slices,
e2e, lint x2, docker tests, and uv-lockfile-check all paid it, all for
identical bytes. Each hop was also a failure mode — the 2026-07-28
slice-5 incident was a transient setup-uv manifest fetch failing a whole
job, and pinning the version narrowed that window without closing it.
hermes-agent-ci-infra now bakes ripgrep 15.1.0, uv 0.9.28, and CPython
3.11 into nousresearch/nous-gke-runner (same versions, so this is a move
not an upgrade), so these steps are pure overhead. Remove them.
The wheel cache is the one part of setup-uv still worth having: it is
per-workspace, not per-image, and without it `uv sync` re-downloads and
re-builds every wheel — the toolchain would be faster to set up and the
sync dramatically slower, a net loss. Replace `enable-cache: true` with
a small .github/actions/uv-cache composite doing the same actions/cache
on ~/.cache/uv, keyed on pyproject.toml + uv.lock. runner.arch is in the
key because the cache holds built wheels and docker.yml runs on arm64
too; the restore-keys prefix means a stale hit still saves most of the
download, and `uv sync --locked` re-resolves from uv.lock regardless so
a partial hit cannot produce a wrong environment.
lint.yml and uv-lockfile-check.yml only `uv tool install` / `uv lock
--check` and never build a project venv, so they drop the setup step
without needing the cache action at all.
Verified against the built image, running as the `runner` user with
`--network none` so nothing can silently re-download: rg 15.1.0, uv
0.9.28, and `uv python find 3.11` all resolve. With hermes-agent's real
pyproject.toml and uv.lock and no setup step of any kind, `uv sync
--locked --python 3.11 --extra dev` completes in 3s into a working
3.11.14 venv. actionlint is clean (the remaining arc-runner-set and
SC2016 warnings are pre-existing on main).
Depends on the image change landing first: pods pull :latest on start,
so merging this before the image is pushed breaks every runner.
The PR-read-only hardening (1d5eb3bf4) gated the WIF auth step on
non-PR events, which also silently gated the Artifact Registry docker
login that step fed. The comment said "PRs read the cache via the pod's
GKE Workload Identity", but pod WI is not ambient for buildx: with no
login, cache-from does an anonymous pull against us-central1-docker.pkg.dev
and gets 403 Forbidden (visible as "failed to configure registry cache
importer" in every PR build since), so every PR built cache-cold
(~12-16 min instead of ~2-3).
Fix: on PR events, mint an access token from the pod's GKE metadata
server (the runner pod's KSA impersonates gha-buildx-cache-ro@, which
has only artifactregistry.reader) and feed it to the same docker login
step. The security boundary is unchanged — PR builds still cannot write
cache layers; cache-to remains gated on the WIF token that only exists
on trusted main-push/release contexts.
Address review findings on the ARC migration:
- docker.yml: WIF auth (and therefore Artifact Registry cache WRITES)
now only runs on non-PR events. The build job runs PR-controlled code
and the publish job reads the same buildcache ref, so a PR-writable
cache was a layer-poisoning vector. PRs of any origin keep cache
READS via the runner pod's GKE Workload Identity — that's where the
15min -> 2-3min win comes from; main pushes repopulate writes.
- docker.yml: profile label is now docker-tests-<arch>. Both matrix
legs uploaded resource-profile-docker-tests; upload-artifact v4+
rejects the duplicate and continue-on-error swallowed it, silently
dropping one arch's profile.
- actions/profile: run the wrapped command with bash -eo pipefail to
match normal `run:` step semantics (a failing `source .venv/...`
must fail the step, not fall through).
- js/e2e/site workflows: bake node22 into the node_modules cache keys
so a future node-version bump can't restore stale native builds
(node-pty, electron postinstall) against an unchanged lockfile.
- test_container_restart_stale_pid: forward deadline_s/interval_s to
wait_for_log instead of silently dropping them.
- doctor.py: refresh a stale comment on the in-container docker branch.
Replace the runner's tests/docker auto-cap heuristic with explicit
width control:
- run_tests.sh forwards HERMES_TEST_WORKERS through its hermetic env -i
(previously silently stripped — the documented override never worked
through the wrapper)
- run_tests_parallel.py drops the _DOCKERD_BOUND_JOBS special-case; the
suite-specific knowledge moves to the one place that runs that suite
- docker.yml pins HERMES_TEST_WORKERS=8. Width sweep with prewarmed
image + split files: -j4 58-62s, -j8 39s, -j12 35s w/ ~2x per-file
contention inflation; 8 is the knee.
Chain verified end-to-end: env var reaches the runner (6-worker probe),
no cap message on docker-only file lists, full suite 53/53 in 37.8s
at -j8.
The docker suite's wall time was max(whale files): four files each
serialized 2-3 ~110s container boots internally while 21 fast files
finished in seconds (P50 9.9s vs max 341s on the ARC runners). The
per-file parallel runner can only overlap what lives in separate files.
- test_dashboard.py -> 3 files (one boot each); shared _http_probe
helper moves to conftest
- test_container_restart.py -> 2 files (restart_container fixture
travels via the shared header; per-file container isolation is the
point of the split)
- test_docker_exec_privilege_drop.py -> boot-heavy e2e login test split
out; the two fast tests stay together
- test_config_migration.py: single test, unchanged
53 tests before and after, zero assertions changed — pure file
reorganization. Local (-j4, same cap as CI): 374.6s -> 58.0s wall,
slowest file 341.7s -> 18.6s.
docker.yml now requests id-token: write for WIF cache auth, but it is
invoked as a reusable workflow from ci.yml on PRs — and a called
workflow cannot request a permission its caller lacks. That mismatch
is a startup_failure (run 30596417013 died before any job). Add
id-token: write to ci.yml's top-level permissions.
Swap type=gha buildx cache for type=registry against
us-central1-docker.pkg.dev/.../ci-cache — same region as the ARC
runners, so layer blobs stop round-tripping to GitHub's cache CDN on
every build/rerun.
Auth is keyless both ways: reads ride the runner pod's GKE Workload
Identity (no login needed for cache-from); writes exchange the
workflow's GitHub OIDC token via WIF (google-github-actions/auth,
fork-guarded — fork PRs build cache-cold exactly like type=gha).
publish keeps its own unconditional WIF auth (trusted contexts only).
Infra: hermes-agent-ci-infra ef8dfb2.
1. test_tui_gateway_server: the two _start_agent_build tests waited only
for the _make_agent 'built' event, then popped the session while
_build's tail was still running. The tail's session.info/error emit
then landed on whatever _real_stdout a LATER test had patched in —
the write_json concurrency test intermittently saw 9 lines instead
of its own 8 (2-in-5 repro locally). Join the build thread (exposed
as session['_agent_build_thread']) before popping. 10/10 clean
full-file runs post-fix, was 2/5 failing.
2. docker.yml: the profile-action conversion dropped the step env —
including HERMES_TEST_IMAGE, so all 25 per-file subprocesses each
docker-built the image inside dind concurrently. That is the root
cause of the 15-minute docker jobs and the teardown timeout storms.
Restore it plus the blank-API-key policy vars.
Root cause of the last two amd64 docker failures: attempt 1 of a flaky
file times out mid-teardown (busy dind), the stale hermes-test-* name
survives, and the file-retry's docker run fails with a name Conflict —
so the retry mechanism itself was poisoned. The fixture now removes the
name BEFORE the test (fresh subprocess retry gets a clean slate), and
teardown swallows a slow-daemon TimeoutExpired instead of erroring a
passing test (1 passed, 1 error -> 1 passed).
Move the dockerd-bound worker cap from a workflow env var into
run_tests_parallel.py: when every file in the run is under
tests/docker/, cap -j at 4 (the suite shares one docker daemon; width
beyond that thrashes it — files stretch ~100s -> ~900s and teardown
docker-rm calls blow their 10s timeout). Explicit -j or
HERMES_TEST_WORKERS always wins; mixed file lists are unaffected.
Verified: docker-only list caps 32->4, -j 12 respected, mixed list
uncapped. Drops the HERMES_TEST_WORKERS=4 pin from docker.yml.
Diagnosis from the profiler + runner logs: with the cgroup-aware
default (-j 16 on the 8-CPU pods) all 25 files run concurrently against
the single dind daemon. Every file stretches to ~900s wall (P50 892s,
CPU-wall 14549s vs 918s wall) and teardown docker-rm calls exceed their
10s timeout — the job dies on teardown errors while tests themselves
pass 53/53. The arm64 lane (2 CPU → -j 4) went green for exactly this
reason. Pin the amd64 lane to the same effective width.
- doctor: 'inside a container' branch rewrote terminal_env to local for
EVERY non-docker backend, so TERMINAL_ENV=vercel_sandbox diagnostics
vanished when doctor ran inside a container (CI runner pods). Scope
the informational skip to the implicit local case only; remote
backends keep their real diagnostics. Fixes
test_doctor_reports_vercel_backend_diagnostics on ARC runners — and
for actual users running doctor in the Docker distribution with a
remote terminal backend configured.
- termux audio test: detect_audio_environment() probes the real host
for containment; pin is_container=False (a Termux device is never a
container) so the containerized runner doesn't flip available=False.
os.cpu_count() reports the HOST cores. In an ARC runner pod
(limit 8 CPU on a 22-core node) the runner spawned -j 44 workers on
8 usable CPUs — ~5x oversubscription. Every 'timing flake' family on
the self-hosted runners (docker rm teardown TimeoutExpired x111,
compression fork, termux probe, pty reaper, session hygiene) is CPU
starvation from that oversubscription, not real test bugs.
Read cgroup v2 cpu.max (v1 cfs_quota fallback) and clamp to host count.
Verified: --cpus=8 container reports 8, bare host unchanged.
HERMES_TEST_WORKERS override still wins.
- test_supports_systemd_services_returns_true_when_systemctl_present:
pin is_container=False (host contract; CI runner pods are containers)
- test_systemd_restart_gracefully_restarts_running_service_and_waits:
stub _preflight_user_systemd — no user D-Bus in runner pods; the test
asserts restart choreography, not D-Bus reachability
Test fixes for ARC runner pods (containers) vs GHA ubuntu-latest VMs:
- test_gateway_wsl / test_copilot_acp_client: pin is_container=False on
host-behavior tests — runner pods ARE containers and the prod code
intentionally behaves differently there. Also drop inherited
HERMES_REAL_HOME so nix dev shells don't leak into the assert.
- honcho memo + skill-utils external-dirs cache: add st_size to the
mtime_ns cache keys. overlayfs (runner pods) coalesces rapid writes
into one mtime tick, so same-tick edits were served stale (3 honcho
pin tests + skill cache invalidation test).
Verified with KUBERNETES_SERVICE_HOST set to simulate the pod env.
Swap all `runs-on: ubuntu-latest` to `runs-on: arc-runner-set` all jobs.
The ARM docker build job in docker.yml uses `${{ matrix.runner }}`
and is left untouched since the GKE runner pool is x86_64 only.
Runners are backed by ARC (Actions Runner Controller) on a GKE cluster
with a spot preemptible node pool that scales based on job demand.
Use the baked Electron dependencies for the desktop E2E job.
The token in the input line is the whole receipt. Drop the notices that
duplicated it somewhere the user was not looking: the drag-drop and
clipboard sys() lines, and the attachedImageNotice / "detected file: X"
activity rows above the status bar.
attachedImageNotice and imageTokenMeta have no callers left.
Every attach path now drops an `[[ Image N ]]` token where you are typing:
drag-drop, clipboard (bracketed and hotkey), /image, /paste. The composer
owns clipboard attach directly instead of calling back out to useMainApp.
Deleting the token is how you unattach — there is no second control.
updateInput is the one choke point every keystroke passes through, so
syncTokens reconciles there and detaches anything erased. That also fixes
a stale image riding along on the next unrelated turn.
Tokens and the input line get refs alongside state: paste-then-immediately
-Enter submits before React has re-rendered, and the submit path has to see
the token that was just added.
A collapsed paste and an attached image are the same idea: a `[[ … ]]`
marker sitting in the input line that stands in for a payload resolved at
submit. Model both as ComposerToken and give them one expander.
Image tokens resolve to nothing — the gateway already holds the file in
attached_images — so expandTokens eats an adjacent space to avoid leaving
a gap mid-sentence. nextImageIndex never reuses an index after a delete,
or two files would collide on one label.
Select-all + Cut emptied the text and left the composer blank — no draft,
no prompt. Delete had the same hole.
The placeholder is painted on `:empty`, and a cleared editor keeps a
scaffolding <br> so the contenteditable can't collapse to a sliver. Those
two facts collide: the moment the break lands the editor has a child,
`:empty` goes false, and the prompt never comes back.
CSS can't infer emptiness on its own either. A text node is invisible to
selectors, so `one<br>` and a lone `<br>` are the same shape — a structural
rule like `:has(> br:only-child)` paints the placeholder straight over the
user's text. The code that empties the editor is what knows, so it marks
the root and the condition reads `:is(:empty, [data-empty])`.
Both writers that reshape that root maintain the marker through one helper:
the normalizer, and renderComposerContents for a restored draft or an undo.
The message-edit composer shares the slot and the rule, so it takes the
same shared class instead of drifting on its own copy.
#74815 fixed the draft this stashed; the placeholder is a separate seam.
The tab strip decided the close gesture from the `uncloseable` flag, which the
workspace sets to keep its pane in the tree — so the one tab whose close now
does something couldn't be ⌘-clicked or middle-clicked, and its right-click
menu had no Close.
Read the gesture off the pane's registered closer instead, with the workspace
registering closeWorkspaceTab. An atom rather than a lookup, since that closer
comes from a wiring effect that lands after the strip's first paint.
The workspace pane can't leave the tree, so "close the main tab" only ever had
one answer wired: shift the next stacked session in. With main as the only tab
there was nothing to shift and ⌘W dead-ended on the tab the user was looking
at.
closeWorkspaceTab is now the one answer for every entry point — stacked
session still wins, and with nothing stacked main drops to a fresh New session
draft. A blank draft and a full-page view stay no-ops: a blank draft already
IS the post-close state.
Chromium on Windows and Linux answers a middle press inside a scroller by
starting the autoscroll pan, and the mouseup that ends the pan never becomes
an auxclick. Every surface carrying the gesture — tab strips, the session
list, the terminal rail — is a scroller, so middle-click only ever worked on
macOS, where autoscroll doesn't exist.
Arm on pointerdown, spend on the pointerup over the same element (press one
tab, release on another and nothing happens), and cancel the middle mousedown
on every press so the pan widget can't appear on a surface that owns the
button. One helper, four call sites.
⌥-drag is the app's force-selection gesture over mouse-mode TUIs, but
xterm's default alt-click-moves-cursor claims the same click and emits one
cursor left/right escape per column of travel. Shells that don't consume
them echo the raw `^[[D` burst into the buffer. One gesture, one meaning.
Both of xterm's link paths activate through `window.open()`, which the
window's setWindowOpenHandler denies, so ⌘-clicking a URL did nothing but
log "Opening link blocked as opener could not be cleared" — and the OSC 8
path fronted that dead end with a raw confirm() dialog. Route both through
the desktop bridge, the path every other external link in the app takes.
⌘-click on macOS, Ctrl-click elsewhere, matching VS Code's integrated
terminal, Terminal.app, and iTerm2. A bare click stays with the selection so
a misclick on a URL can't launch a browser.
The two exact-list assertions in test_tui_gateway_server froze the toolset
list and broke the moment _RECENTLY_SHIPPED_TOOLSETS back-filled bfl onto a
saved platform list — the exact behavior the sibling change ships on purpose.
Assert the invariant instead: the expected base set is present, and anything
extra must be inside _RECENTLY_SHIPPED_TOOLSETS (vacuously exact again once
that set empties between releases).
send_video got the 60s read_timeout but send_voice/send_audio/send_photo/
send_document/send_media_group/send_animation upload through the same PTB
request path and hit the same server-side processing wait before the
response arrives. Same class, all sites: they all pass
_MEDIA_SEND_READ_TIMEOUT now. Also drops an unused test helper.
