Replace the runner's tests/docker auto-cap heuristic with explicit
width control:
- run_tests.sh forwards HERMES_TEST_WORKERS through its hermetic env -i
(previously silently stripped — the documented override never worked
through the wrapper)
- run_tests_parallel.py drops the _DOCKERD_BOUND_JOBS special-case; the
suite-specific knowledge moves to the one place that runs that suite
- docker.yml pins HERMES_TEST_WORKERS=8. Width sweep with prewarmed
image + split files: -j4 58-62s, -j8 39s, -j12 35s w/ ~2x per-file
contention inflation; 8 is the knee.
Chain verified end-to-end: env var reaches the runner (6-worker probe),
no cap message on docker-only file lists, full suite 53/53 in 37.8s
at -j8.
The docker suite's wall time was max(whale files): four files each
serialized 2-3 ~110s container boots internally while 21 fast files
finished in seconds (P50 9.9s vs max 341s on the ARC runners). The
per-file parallel runner can only overlap what lives in separate files.
- test_dashboard.py -> 3 files (one boot each); shared _http_probe
helper moves to conftest
- test_container_restart.py -> 2 files (restart_container fixture
travels via the shared header; per-file container isolation is the
point of the split)
- test_docker_exec_privilege_drop.py -> boot-heavy e2e login test split
out; the two fast tests stay together
- test_config_migration.py: single test, unchanged
53 tests before and after, zero assertions changed — pure file
reorganization. Local (-j4, same cap as CI): 374.6s -> 58.0s wall,
slowest file 341.7s -> 18.6s.
docker.yml now requests id-token: write for WIF cache auth, but it is
invoked as a reusable workflow from ci.yml on PRs — and a called
workflow cannot request a permission its caller lacks. That mismatch
is a startup_failure (run 30596417013 died before any job). Add
id-token: write to ci.yml's top-level permissions.
Swap type=gha buildx cache for type=registry against
us-central1-docker.pkg.dev/.../ci-cache — same region as the ARC
runners, so layer blobs stop round-tripping to GitHub's cache CDN on
every build/rerun.
Auth is keyless both ways: reads ride the runner pod's GKE Workload
Identity (no login needed for cache-from); writes exchange the
workflow's GitHub OIDC token via WIF (google-github-actions/auth,
fork-guarded — fork PRs build cache-cold exactly like type=gha).
publish keeps its own unconditional WIF auth (trusted contexts only).
Infra: hermes-agent-ci-infra ef8dfb2.
1. test_tui_gateway_server: the two _start_agent_build tests waited only
for the _make_agent 'built' event, then popped the session while
_build's tail was still running. The tail's session.info/error emit
then landed on whatever _real_stdout a LATER test had patched in —
the write_json concurrency test intermittently saw 9 lines instead
of its own 8 (2-in-5 repro locally). Join the build thread (exposed
as session['_agent_build_thread']) before popping. 10/10 clean
full-file runs post-fix, was 2/5 failing.
2. docker.yml: the profile-action conversion dropped the step env —
including HERMES_TEST_IMAGE, so all 25 per-file subprocesses each
docker-built the image inside dind concurrently. That is the root
cause of the 15-minute docker jobs and the teardown timeout storms.
Restore it plus the blank-API-key policy vars.
Root cause of the last two amd64 docker failures: attempt 1 of a flaky
file times out mid-teardown (busy dind), the stale hermes-test-* name
survives, and the file-retry's docker run fails with a name Conflict —
so the retry mechanism itself was poisoned. The fixture now removes the
name BEFORE the test (fresh subprocess retry gets a clean slate), and
teardown swallows a slow-daemon TimeoutExpired instead of erroring a
passing test (1 passed, 1 error -> 1 passed).
Move the dockerd-bound worker cap from a workflow env var into
run_tests_parallel.py: when every file in the run is under
tests/docker/, cap -j at 4 (the suite shares one docker daemon; width
beyond that thrashes it — files stretch ~100s -> ~900s and teardown
docker-rm calls blow their 10s timeout). Explicit -j or
HERMES_TEST_WORKERS always wins; mixed file lists are unaffected.
Verified: docker-only list caps 32->4, -j 12 respected, mixed list
uncapped. Drops the HERMES_TEST_WORKERS=4 pin from docker.yml.
Diagnosis from the profiler + runner logs: with the cgroup-aware
default (-j 16 on the 8-CPU pods) all 25 files run concurrently against
the single dind daemon. Every file stretches to ~900s wall (P50 892s,
CPU-wall 14549s vs 918s wall) and teardown docker-rm calls exceed their
10s timeout — the job dies on teardown errors while tests themselves
pass 53/53. The arm64 lane (2 CPU → -j 4) went green for exactly this
reason. Pin the amd64 lane to the same effective width.
- doctor: 'inside a container' branch rewrote terminal_env to local for
EVERY non-docker backend, so TERMINAL_ENV=vercel_sandbox diagnostics
vanished when doctor ran inside a container (CI runner pods). Scope
the informational skip to the implicit local case only; remote
backends keep their real diagnostics. Fixes
test_doctor_reports_vercel_backend_diagnostics on ARC runners — and
for actual users running doctor in the Docker distribution with a
remote terminal backend configured.
- termux audio test: detect_audio_environment() probes the real host
for containment; pin is_container=False (a Termux device is never a
container) so the containerized runner doesn't flip available=False.
os.cpu_count() reports the HOST cores. In an ARC runner pod
(limit 8 CPU on a 22-core node) the runner spawned -j 44 workers on
8 usable CPUs — ~5x oversubscription. Every 'timing flake' family on
the self-hosted runners (docker rm teardown TimeoutExpired x111,
compression fork, termux probe, pty reaper, session hygiene) is CPU
starvation from that oversubscription, not real test bugs.
Read cgroup v2 cpu.max (v1 cfs_quota fallback) and clamp to host count.
Verified: --cpus=8 container reports 8, bare host unchanged.
HERMES_TEST_WORKERS override still wins.
- test_supports_systemd_services_returns_true_when_systemctl_present:
pin is_container=False (host contract; CI runner pods are containers)
- test_systemd_restart_gracefully_restarts_running_service_and_waits:
stub _preflight_user_systemd — no user D-Bus in runner pods; the test
asserts restart choreography, not D-Bus reachability
Test fixes for ARC runner pods (containers) vs GHA ubuntu-latest VMs:
- test_gateway_wsl / test_copilot_acp_client: pin is_container=False on
host-behavior tests — runner pods ARE containers and the prod code
intentionally behaves differently there. Also drop inherited
HERMES_REAL_HOME so nix dev shells don't leak into the assert.
- honcho memo + skill-utils external-dirs cache: add st_size to the
mtime_ns cache keys. overlayfs (runner pods) coalesces rapid writes
into one mtime tick, so same-tick edits were served stale (3 honcho
pin tests + skill cache invalidation test).
Verified with KUBERNETES_SERVICE_HOST set to simulate the pod env.
Swap all `runs-on: ubuntu-latest` to `runs-on: arc-runner-set` all jobs.
The ARM docker build job in docker.yml uses `${{ matrix.runner }}`
and is left untouched since the GKE runner pool is x86_64 only.
Runners are backed by ARC (Actions Runner Controller) on a GKE cluster
with a spot preemptible node pool that scales based on job demand.
Use the baked Electron dependencies for the desktop E2E job.
The token in the input line is the whole receipt. Drop the notices that
duplicated it somewhere the user was not looking: the drag-drop and
clipboard sys() lines, and the attachedImageNotice / "detected file: X"
activity rows above the status bar.
attachedImageNotice and imageTokenMeta have no callers left.
Every attach path now drops an `[[ Image N ]]` token where you are typing:
drag-drop, clipboard (bracketed and hotkey), /image, /paste. The composer
owns clipboard attach directly instead of calling back out to useMainApp.
Deleting the token is how you unattach — there is no second control.
updateInput is the one choke point every keystroke passes through, so
syncTokens reconciles there and detaches anything erased. That also fixes
a stale image riding along on the next unrelated turn.
Tokens and the input line get refs alongside state: paste-then-immediately
-Enter submits before React has re-rendered, and the submit path has to see
the token that was just added.
A collapsed paste and an attached image are the same idea: a `[[ … ]]`
marker sitting in the input line that stands in for a payload resolved at
submit. Model both as ComposerToken and give them one expander.
Image tokens resolve to nothing — the gateway already holds the file in
attached_images — so expandTokens eats an adjacent space to avoid leaving
a gap mid-sentence. nextImageIndex never reuses an index after a delete,
or two files would collide on one label.
Select-all + Cut emptied the text and left the composer blank — no draft,
no prompt. Delete had the same hole.
The placeholder is painted on `:empty`, and a cleared editor keeps a
scaffolding <br> so the contenteditable can't collapse to a sliver. Those
two facts collide: the moment the break lands the editor has a child,
`:empty` goes false, and the prompt never comes back.
CSS can't infer emptiness on its own either. A text node is invisible to
selectors, so `one<br>` and a lone `<br>` are the same shape — a structural
rule like `:has(> br:only-child)` paints the placeholder straight over the
user's text. The code that empties the editor is what knows, so it marks
the root and the condition reads `:is(:empty, [data-empty])`.
Both writers that reshape that root maintain the marker through one helper:
the normalizer, and renderComposerContents for a restored draft or an undo.
The message-edit composer shares the slot and the rule, so it takes the
same shared class instead of drifting on its own copy.
#74815 fixed the draft this stashed; the placeholder is a separate seam.
The tab strip decided the close gesture from the `uncloseable` flag, which the
workspace sets to keep its pane in the tree — so the one tab whose close now
does something couldn't be ⌘-clicked or middle-clicked, and its right-click
menu had no Close.
Read the gesture off the pane's registered closer instead, with the workspace
registering closeWorkspaceTab. An atom rather than a lookup, since that closer
comes from a wiring effect that lands after the strip's first paint.
The workspace pane can't leave the tree, so "close the main tab" only ever had
one answer wired: shift the next stacked session in. With main as the only tab
there was nothing to shift and ⌘W dead-ended on the tab the user was looking
at.
closeWorkspaceTab is now the one answer for every entry point — stacked
session still wins, and with nothing stacked main drops to a fresh New session
draft. A blank draft and a full-page view stay no-ops: a blank draft already
IS the post-close state.
Chromium on Windows and Linux answers a middle press inside a scroller by
starting the autoscroll pan, and the mouseup that ends the pan never becomes
an auxclick. Every surface carrying the gesture — tab strips, the session
list, the terminal rail — is a scroller, so middle-click only ever worked on
macOS, where autoscroll doesn't exist.
Arm on pointerdown, spend on the pointerup over the same element (press one
tab, release on another and nothing happens), and cancel the middle mousedown
on every press so the pan widget can't appear on a surface that owns the
button. One helper, four call sites.
⌥-drag is the app's force-selection gesture over mouse-mode TUIs, but
xterm's default alt-click-moves-cursor claims the same click and emits one
cursor left/right escape per column of travel. Shells that don't consume
them echo the raw `^[[D` burst into the buffer. One gesture, one meaning.
Both of xterm's link paths activate through `window.open()`, which the
window's setWindowOpenHandler denies, so ⌘-clicking a URL did nothing but
log "Opening link blocked as opener could not be cleared" — and the OSC 8
path fronted that dead end with a raw confirm() dialog. Route both through
the desktop bridge, the path every other external link in the app takes.
⌘-click on macOS, Ctrl-click elsewhere, matching VS Code's integrated
terminal, Terminal.app, and iTerm2. A bare click stays with the selection so
a misclick on a URL can't launch a browser.
The two exact-list assertions in test_tui_gateway_server froze the toolset
list and broke the moment _RECENTLY_SHIPPED_TOOLSETS back-filled bfl onto a
saved platform list — the exact behavior the sibling change ships on purpose.
Assert the invariant instead: the expected base set is present, and anything
extra must be inside _RECENTLY_SHIPPED_TOOLSETS (vacuously exact again once
that set empties between releases).
send_video got the 60s read_timeout but send_voice/send_audio/send_photo/
send_document/send_media_group/send_animation upload through the same PTB
request path and hit the same server-side processing wait before the
response arrives. Same class, all sites: they all pass
_MEDIA_SEND_READ_TIMEOUT now. Also drops an unused test helper.
ChatView migrated tip-keyed composer queue/draft entries onto queueSessionKey
whenever the two ids differed. queueSessionKey is route-driven and can flip to
Session B a frame before the store selection leaves Session A, so migrate
re-homed A queue entries onto B and the idle ChatBar auto-drained them into
the wrong chat.
Gate migrate on same-conversation lineage only (tip to root). Also honor
lineage when background queue drain decides selected/busy, so a root queue key
is not treated as idle/offscreen while the compression tip is still working.
The row fell back to the global $repoStatus whenever repoPath was blank,
painting the main pane's branch and ± onto a tile whose cwd hadn't
resolved yet. The fallback bought nothing — the primary computed is keyed
to $currentCwd, which is empty in exactly that case — and cost a rail
showing a tree the session was never in.
