Port from Kilo-Org/kilocode#12508: a stray read-only state.db / -wal /
-shm (sudo run, restored backup, copied dotfiles) previously killed
SessionDB init with an opaque 'sqlite3.OperationalError: attempt to
write a readonly database' raised from deep inside _init_schema —
naming no file and no fix — and the obvious wrong 'fix' (deleting the
-wal) silently loses committed transactions.
New preflight_db_writability() runs before the first connection on both
DB open paths (SessionDB.__init__ and hermes_cli.kanban_db.connect):
- files inside the Hermes home tree are repaired with chmod u+rw (the
safe scope: Hermes owns them, and the OS makes chmod fail on files
the user doesn't own, which bounds the repair exactly);
- anything else (root-owned files, read-only mounts, custom paths)
fails fast with an error naming the exact file and the exact chmod
command, plus an explicit 'do NOT delete the -wal' warning;
- WAL sidecars are never deleted or truncated — once writable, the
normal open path checkpoints committed frames into the DB.
Proven live on main first: chmod 444 state.db -> SessionDB() raises the
opaque readonly error. With the fix: in-home DBs self-heal; out-of-home
DBs get the actionable message. Sabotage run confirms the integration
tests fail without the wiring (2 failed / 10 passed).
A long-lived gateway can have platform routing (HERMES_SESSION_* /
HERMES_CRON_AUTO_DELIVER_*) mirrored in os.environ from a previous turn.
_default_spawn() copied that process environment verbatim into detached
kanban workers, so a worker calling kanban_create treated the inherited
chat/topic as its origin and auto-subscribed the child task — the task's
terminal notification then woke an unrelated chat.
Strip every registered session-context routing key from the worker env
unconditionally (the dispatcher is detached from every conversation);
board, workspace, task, branch, profile, model, and credential
propagation are unchanged.
Salvaged from PR #69181 (both commits squashed; the PR's second commit
fixed the first's engagement-latch assumption).
Fixes the boot-storm half of issue #29905: kanban_notify_subs.last_event_id
defaulted to 0, so a subscription created on an already-active task replayed
the task's ENTIRE terminal-event backlog on the next notifier tick. With
many stale subs (27 observed in the report) a gateway boot after downtime
burst 100+ notifications in one go.
add_notify_sub now snaps the cursor to COALESCE(MAX(task_events.id), 0) for
the task inside the same INSERT, so new subscriptions start caught up and
only receive events that occur AFTER subscribing. The gateway slash-command
and kanban-tool auto-subscribe paths run at task creation, where the
snapshot is just the 'created' event — behavior there is unchanged.
Stale fixtures that asserted the literal 0 creation cursor now assert
'cursor unchanged/unclaimed' instead, which is what they actually meant.
Salvaged from PR #63001 (reduced scope): probe each board with the new
read-only kanban_db.count_notify_subs() before the writable connect(),
so boards with zero subscriptions are never opened writable on the 5s
notifier tick (no schema migration, no WAL/-shm sidecar churn, no
checkpoints).
The PR's machine-global .notifier.lock singleton gate was deliberately
NOT salvaged: a lock-winning default-profile gateway cannot deliver a
secondary profile's subscriptions in standalone-profile deployments
(profile routing fails closed in _authorization_adapter), so the lock
could suppress delivery entirely. The probe captures the per-tick cost
win without that regression.
Copy gateway notification subscriptions from parent tasks to child tasks created by create_task(..., parents=...), link_tasks(), and decompose_triage_task().
Inherited subscriptions start at the child's current event cursor, so linking an existing child does not replay pre-link task events.
Addresses review findings on the previous commits. Three of them were real
defects I reproduced against my own head before fixing.
1. Check/use race (BLOCKING). read_header_bytes_preopen() checked
has_live_connection() under _live_lock, released it, then did the raw
open/read/close outside the lock; connect_tracked() opened before
registering. A thread could pass the "nothing is live" check, another
could open a connection and BEGIN IMMEDIATE, and the first thread's
close() then cancelled its POSIX locks -- the exact bug this guard
exists to prevent. Reproduced deterministically (BLOCKED -> ACQUIRED).
_live_lock now spans all three lifecycle transitions: open+register,
unregister+close, and check+open+read+close.
2. Read-only connections keyed by URI spelling (BLOCKING). SessionDB's
read-only path opens file:/…/state.db?mode=ro; that string was fed to
Path.resolve(), producing <cwd>/file:/…/state.db?mode=ro. No probe of
the real Path could match, so read-only connections were invisible to
the guard and their locks cancellable. Reproduced with no forced
scheduling. Keys now come from PRAGMA database_list (canonical path),
with an explicit tracking_path override.
3. Fail-open wrapper (HIGH). _connect_tracked_db() caught every exception
and retried an untracked plain connect, so any error silently disabled
the guard. Now only ImportError (scaffold installs without hermes_cli)
falls back; real failures propagate.
4. Backup paths that warned and proceeded (MEDIUM). _backup_corrupt_db()
and _backup_db_file() raw-read live databases; they now REFUSE when a
connection is live rather than warning. Losing a forensic copy beats
corrupting the database being rescued.
Custom factories are no longer rejected (that broke legitimate callers) nor
silently untracked -- the tracking close() is mixed into whatever factory is
in play, including when an opener substitutes its own after the fact.
WAL POLICY: #70055 is RESTORED, not reverted. My earlier justification was
confounded -- the clean WAL result came from 3.53.1, which carries both the
WAL-reset fix AND 3.51.0's broken-lock defenses, so it said nothing about the
bundled 3.50.4. Re-measured on 3.50.4 with the lock fix in place: WAL 0/3 and
DELETE 0/3, i.e. no evidence WAL is safer. Upstream still documents the
WAL-reset bug through 3.51.2 as serious. Keeping new databases out of WAL
until a fixed runtime ships is the conservative call, and the WAL policy does
not belong in this root-cause fix.
Six sabotage runs confirm each new test fails when its defect is reinstated
(including two that initially did NOT -- the race test was rewritten to pause
inside the byte read, and a separate test added for the opener-substituted
factory path). 1124 targeted tests green.
Completeness pass over the previous commit.
Registry leak (would have silently disabled the guard): the first version
incremented on open but decremented only in SessionDB.close(). kanban's
connect() hands raw connections to callers who close them directly (4 sites),
so its counter only ever went up — after enough kanban operations every
byte-probe on that path would be refused forever, disabling zeroed-file and
header detection. Replaced manual track/untrack with a TrackedConnection
subclass that untracks in close(), the one method every close path goes
through. Verified across plain close, contextlib.closing, double close,
nested lifetimes, and 100-cycle churn; `with conn:` (a transaction scope, not
a close) correctly stays tracked.
Also: a caller-supplied factory now wins instead of raising TypeError on a
duplicate kwarg, since tracking is an optimisation for the probe guard, not a
precondition for opening the database.
Removed _apply_delete_for_wal_reset_bug, dead after the force-DELETE revert.
Audit notes:
- DELETE mode is still reachable via the NFS/SMB/FUSE fallback and remains
correct there; those users are protected by the raw-read fix, not by the
journal mode.
- The remaining whole-file reads are on genuinely offline artifacts:
_backup_db_file (DB won't open; bytes preserved for forensics),
_backup_corrupt_db (quarantine path, now warns if a connection is live),
and backup verification of snapshots. backup._safe_copy_db already uses
the SQLite backup API rather than a byte copy.
- The mechanism is POSIX-specific (Windows byte-range locks are handle-scoped,
not process-scoped), so this is a Linux/macOS correctness fix; the change is
platform-neutral and safe on Windows.
Sibling tests updated: session recovery no longer expects a DELETE-mode
recovered DB, and the kanban WAL-fallback test patches the connect site that
actually runs now.
`hermes sessions optimize` could corrupt state.db. Root cause is Hermes,
not the SQLite WAL-reset bug (#69784).
close() on ANY file descriptor for a SQLite database cancels every POSIX
advisory lock the process holds on that file, including a running VACUUM's
EXCLUSIVE lock (sqlite.org/howtocorrupt.html section 2.2). Hermes byte-probed
live databases in several hot paths: the zeroed-state.db detector runs on every
SessionDB construction (and the gateway builds those constantly), and kanban's
post-commit invariant check ran after every COMMIT. While VACUUM rewrote the
file, those probes dropped its lock and let other processes write into it.
A/B against the real code, only variable being the raw read:
SQLite 3.50.4, VACUUM + concurrent writers, DELETE mode
raw open/close during VACUUM 8 vacuums, 319 vacuum errors, 2/2 corrupt
no raw read (control) 229 vacuums, 0 vacuum errors, 0/2 corrupt
SQLite 3.53.1 (WAL-reset FIXED) reproduces identically: 2/2 corrupt.
After this change: 0/4 corrupt, 0 vacuum errors.
Because the upgraded runtime corrupts too, replacing the embedded SQLite does
not fix this class; and because DELETE is where it reproduces, #70055's
"force DELETE on vulnerable builds" mitigation steered users into the failing
mode. That gate is reverted here: vulnerable builds get WAL again and still
warn so operators can upgrade.
- add hermes_cli/sqlite_safe_read.py: read page_count via PRAGMA over the
existing connection instead of open()+seek(28); byte-level probes are
restricted to before any connection exists and refused once one is live,
with an explicit force= escape for offline artifacts (snapshots, archives)
- track live connections in SessionDB and kanban's connect so that guard is
enforced rather than merely documented
- kanban's torn-extend check now only applies under a rollback journal; in WAL
a committed page may still legitimately sit in the -wal file
- revert the force-DELETE WAL gate and update the tests that pinned it
Regression tests assert the behavioural contract (an external process stays
locked out across Hermes' inspection calls) and were verified to fail when the
old raw-open behaviour is restored.
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.
Fixes#53428 (master tracker for Windows GBK locale crash).
Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
Decompose children inherit the root's literal workspace_path (#37172),
so every sibling of a worktree-kind root points at the SAME checkout.
_resolve_worktree_workspace's existing-checkout shortcut then reuses
that directory on whatever branch is currently checked out, ignoring
the task's own branch_name. Net effect: sibling workers — which can be
promoted and dispatched concurrently — run in one directory on the
first sibling's branch, with no lock. Work lands on the wrong task's
branch (provenance corruption) and concurrent siblings trample each
other's index/tree.
Fix, two layers:
- decompose_triage_task: worktree-kind children no longer inherit the
root's literal path; each child materializes its own
<repo>/.worktrees/<child-id> at dispatch (dir/scratch inheritance
unchanged — children legitimately share those).
- _resolve_worktree_workspace: when the requested path is an existing
checkout of a DIFFERENT branch, fall back to a fresh
<repo>/.worktrees/<task-id> instead of silently reusing it (heals
rows that already carry a shared path). Same-branch reuse and the
no-repo/own-path degenerate cases keep the legacy behaviour.
Tests: tests/hermes_cli/test_kanban_worktree_isolation.py (5); full
test_kanban_db.py + test_kanban_decompose_db.py suites pass unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Default kanban_create children now keep fresh scratch paths, while explicit dir sharing remains supported and project context resolves to a per-task worktree. Surface resolved workspace fields in create responses/events and cover scratch mutation, nesting, explicit sharing, and project inheritance.
Fixes#67567
Adds the missing write path for the per-task model_override column (which
was previously only settable via manual SQL) and pairs it with a
provider_override so cross-provider switches resolve correctly:
- kanban_db: provider_override column (+migration), set_model_override()
with model_override_set event, create_task(model_override=,
provider_override=), dispatcher spawns worker with -m <model>
[--provider <name>]
- dashboard: Model row in the task drawer — dropdown fed by a new
/model-options endpoint (build_models_payload substrate, provider-grouped,
free-text fallback), PATCH + bulk model override support
- CLI: kanban create --model/--provider, new kanban set-model subcommand,
show prints the provider
- agent tools: kanban_create accepts model/provider; show/list expose
provider_override
Rate-limit recovery flow: override is settable on running tasks and takes
effect on the next dispatch, without touching the worker profile's config.
Adds kanban_db.repair_db() — a structured, non-raising wrapper around
the same narrow repair policy as the connect-time guard: probe with
PRAGMA integrity_check under the board's cross-process init flock;
quarantine the corrupt bytes FIRST via the content-addressed backup;
REINDEX only when every integrity message is index-scoped; re-check;
report ok / repaired / corrupt / missing. Locked/busy OperationalError
still propagates raw (a locked healthy DB is not corruption and gets
no quarantine), and a repair invalidates the per-process healthy-path
cache so the next connect() re-probes.
The CLI verb reports status human-readably (or --json), exits 0 for
ok/repaired/missing and 1 when the DB is still corrupt (non-index
corruption stays fail-closed with manual-recovery guidance). It
dispatches BEFORE kanban_command's auto-init: init_db() raises
KanbanDbCorruptError on a corrupt board, which previously would have
made a repair verb unreachable on exactly the boards that need it.
CLI tests drive the real argparse surface (build_parser +
kanban_command) against real corrupted SQLite fixtures.
Kanban connections set wal_autocheckpoint=100, but SQLite's passive
autocheckpoint backs off whenever any reader holds an open snapshot —
on a busy multi-process board the -wal file can grow without bound
between gateway restarts.
After each successful dispatch tick, while still holding the board's
single-writer dispatch flock, run PRAGMA wal_checkpoint(TRUNCATE)
best-effort at a coarse interval (>=5 min since this process last
checkpointed that board; module-level per-path monotonic timestamp, so
multi-board dispatchers checkpoint each board on its own clock).
Success and busy/locked skips are both logged at DEBUG; a failing
checkpoint can never fail the tick.
Content-addressed quarantine backups dedupe identical corrupt bytes,
but corruption that keeps mutating between failures (partial repairs,
further damage across dispatcher retries, multi-profile fleets) mints a
new sha-named backup every round — a user accumulated 124
.corrupt.*.bak files with no bound.
After each NEW backup is created, prune oldest-by-mtime backups beyond
_CORRUPT_BACKUP_RETENTION (module constant, default 10), including the
copied -wal/-shm sidecars. The just-created backup is always exempt
(copy2 preserves the source mtime, which can be older than existing
backups). Pruning is best-effort and never masks the corruption error
about to be raised; dedupe of identical corrupt bytes is unchanged.
_guard_existing_db_is_healthy previously failed closed on ANY
integrity_check failure, including the index-scoped class ('wrong # of
entries in index <name>' / 'row N missing from index <name>') where the
table b-trees are intact and REINDEX rebuilds the damaged indexes
losslessly. Boards hit by that class were bricked until manual surgery
even though SQLite can fix them in-place.
Now, when integrity_check output consists ONLY of index-scoped errors
(index name parsed generically from the message — no hardcoded list):
1. quarantine the corrupt bytes FIRST via the existing content-
addressed _backup_corrupt_db,
2. under the caller-held cross-process init flock, REINDEX each named
index (falling back to bare REINDEX if a parsed name doesn't
resolve),
3. re-run integrity_check and proceed only if it comes back clean.
Any non-index error class (page corruption, malformed image, freelist
damage) — or a REINDEX whose re-check is still dirty — fails closed
exactly as before: backup + KanbanDbCorruptError, no silent recreation.
Transient OperationalError (locked/busy) still propagates raw with no
quarantine.
Tests build a real board DB and corrupt a live index via the
writable_schema/partial-index REINDEX trick to produce the genuine
'wrong # of entries in index' shape, then assert auto-repair recovers
with data intact, page corruption still raises, and a dirty re-check
fails closed.
The salvaged attachment-toolset commit predated main centralizing the
25 MB cap as kanban_db.KANBAN_ATTACHMENT_MAX_BYTES and re-introduced a
private _MAX_ATTACHMENT_BYTES alias. Drop the duplicate: kanban_db's
store_attachment_bytes(), the dashboard upload endpoint, and the
kanban_attach_url tool all reference the one shared constant now, and
the tests monkeypatch that same name.
The kanban board has had full attachment storage and a dashboard HTTP
API (upload/list/download/delete) since #35338, but there was no agent
toolset tool and no `hermes kanban` CLI verb for attachments. Agents and
scripts that don't go through the dashboard server (or can't touch the DB
directly) had no way to create or read real attachments — only links in
comments.
Close that gap by mirroring the existing comment surface:
- `kanban_db.store_attachment_bytes()` — one shared write path (validate
name, enforce the 25 MB cap, write the blob under the per-task dir with
collision-free naming, insert the metadata row, clean up an orphan blob
if the insert fails). `_MAX_ATTACHMENT_BYTES`, `_safe_attachment_name`,
and a new `_collision_free_path` move here so the dashboard, the tool,
and the CLI all share one implementation and can't drift.
- Tools (`tools/kanban_tools.py`): `kanban_attach` (inline base64),
`kanban_attach_url` (server-side http/https fetch with the same cap),
`kanban_attachments` (list). Write tools respect worker task-ownership;
list is read-only. Registered in the `kanban` toolset.
- CLI (`hermes_cli/kanban.py`): `attach <id> <path>`, `attachments <id>`,
`attach-rm <attachment_id>`.
- Dashboard `upload_task_attachment` now imports the shared helpers and
uses `_collision_free_path` — behavior identical (still streams to disk
with the cap, still 413 on overflow).
- Docs (AGENTS.md, kanban-worker skill) and toolset membership updated.
Tests: tool round-trip + oversize + bad base64 + ownership; attach_url
against a local HTTP fixture incl. oversize-mid-stream and non-http
scheme rejection; CLI attach/attachments/attach-rm; shared-helper unit
tests; dashboard parity preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address the hermes-sweeper review of #61233: the bounded retry budget
is now a clean-exit-specific streak, not a share of the unified
consecutive_failures counter.
- detect_crashed_workers stamps a protocol_violation marker into the
violation run's metadata (via the event payload _end_run copies);
_protocol_violation_streak derives the streak from run history:
consecutive most-recent violation runs, rate_limited runs neutral
(mirroring their unified-counter treatment), any other closed run
resets it. Mixed failure kinds can neither consume nor extend the
budget.
- Below-budget violations no longer call _record_task_failure at all:
the task returns to ready with last_failure_error stamped directly
(including the corrective retry guidance wording adopted from #61817,
which build_worker_context surfaces to the retry worker) and the
unified counter is untouched, keeping the two budgets independent.
- At the bound the trip funnels through _record_task_failure with a new
keyword-only force_trip=True: the reaper has already resolved the
per-task max_retries override against the violation streak itself, so
the threshold comparison is skipped rather than double-applied.
max_retries keeps its documented top precedence in both directions:
max_retries=1 blocks on the first violation, max_retries=5 blocks on
the fifth consecutive one, unset uses the default bound of 3.
- Replace the first-violation-blocks regression test with five tests:
first occurrence retries (ready + guidance stamped + no gave_up +
unified counter untouched); streak trips exactly at the bound with
protocol_violations/protocol_violation_limit in the gave_up payload
and the auto-blocked side channel set; a prior nonzero crash does not
consume the violation budget; a non-violation failure between
violations resets the streak; max_retries precedence both directions.
All five fail against the previously reviewed diff and pass with this
follow-up. The test harness resolves hermes_cli.kanban_db fresh and
uses that single module object for the exit registry, liveness patch,
and reaper — earlier suite tests reload the module, and the old
mixed-object harness made _classify_worker_exit return unknown (the
reason the old test failed in full-suite runs on main).
Kanban suite: zero introduced failures vs upstream/main tip (62 vs 63
pre-existing environmental failures — the one no longer failing is the
old violation test this replaces; 662 passed vs 657).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A worker that exits 0 without calling kanban_complete/kanban_block
(model stops early, transient tool wedge) tripped the failure breaker
on FIRST occurrence and the task was blocked. These are overwhelmingly
transient: with a bounded retry (limit 3, tracked via a violation
fingerprint) ~96%% of them complete on respawn. Genuine repeat
offenders still trip the breaker at the limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_default_spawn sets HERMES_KANBAN_GOAL_MODE=1 but launched 'chat -q' without
-Q; _run_kanban_goal_loop_q only executes in the quiet single-query branch,
so goal-mode never ran for dispatcher-spawned workers — they got one turn,
printed text, exited rc=0, and tripped the protocol-violation circuit
breaker (2026-06-09, cards t_d9cbe312 et al). Root-cause report + upstream
issue draft in kanban workspace t_720c5c60.
Dragging a task done→ready did nothing: the respawn guard saw a run that
completed within the success window and deferred forever, unable to tell a
deliberate operator re-run from a status flap. Now a re-queue event
(status change, promote, unblock, reclaim) AFTER the completion bypasses
the recent_success guard, so an explicit done→ready runs again.
An inherited HERMES_TUI=1 or a `display.interface: tui` config default sent
kanban workers into the Ink TUI, whose no-TTY bail-out exits 0 without doing
the task — every attempt ended in "protocol violation". Two layers:
- _default_spawn pins `--cli` (highest-precedence interface flag) and strips
HERMES_TUI from the child env (covers older builds on PATH).
- _resolve_use_tui gates ambient TUI prefs (env/config) behind a real TTY;
an explicit --tui still wins so the informative bail-out stays reachable.
Multi-agent boards leak staleness: a sibling worker's parent handoff,
comment, or prior-attempt summary gets read by the next worker as live
truth even when it's a day old. build_worker_context surfaced the text
with (at best) a bare absolute timestamp, which an LLM reads as fact
regardless of age — parent results had no timestamp at all.
Adds a coarse relative-age stamp (just now / 18h ago / 3d ago) to every
recalled-state line and a one-line 'point-in-time snapshot, re-verify
against source' frame on the parent-results section, so the worker sees
when handoffs were produced and re-checks stale ones before acting.
Follow-up to #53791 addressing review feedback: the footgun checker treated
capture_output=/stdout=/stderr=/check_output as proof a subprocess can't pop a
Windows console. That invariant is false — stream redirection controls where a
child's output goes, not whether a console is allocated. From a console-less
parent (Desktop/Electron, pythonw.exe, detached gateway/cron) a console-subsystem
child still flashes a window even when fully captured.
- check-windows-footguns.py: capture/redirect/check_output is no longer a blanket
safe-pass. Added _WINDOWS_FLASHING_PROGRAMS (git/gh/npm/node/python/uv/ffmpeg/
docker/powershell/…); calls to those are flagged even when captured. Non-flashing
programs keep the capture exemption (no 271-site noise). _subprocess_compat.run/
popen calls are inherently safe (wrapper injects CREATE_NO_WINDOW).
- Routed the 35 genuine flashing git/gh/npm/uv/ffmpeg/docker spawns through the
_subprocess_compat.run/popen chokepoint (Brooklyn's wrapper from #53810) — the
durable fix, not per-site annotations. cmd.exe /c start stays # ok (intentional).
- Updated tests + CONTRIBUTING.md rule #17 to the corrected invariant.
* feat(kanban): typed block reasons + unblock-loop breaker
Stops the kanban blocked-task loop: a worker blocks a task, a cron
unblocks it, the worker re-blocks for the same reason, repeat forever.
block_task now takes a typed kind and a persistent block_recurrences
counter on the tasks table:
- kind=dependency routes to todo (parent-gated, auto-resumed), never
the human 'blocked' bucket a cron would keep unblocking.
- needs_input/capability/transient/untyped land in blocked; each
same-cause re-block after an unblock increments block_recurrences,
and at BLOCK_RECURRENCE_LIMIT (default 2) the task routes to triage
for a human instead of blocked.
- unblock_task no longer resets block_recurrences (the amnesia that
let the loop run unbounded); complete_task clears it on success.
Wired through the worker kanban_block tool (new kind arg) and the
hermes kanban block --kind CLI flag, both reporting where the task
actually landed. Docs + 11 new tests; 536 existing kanban tests green.
* test(kanban): make second-block notify test use a distinct block cause
test_notifier_second_blocked_delivers blocked the same task twice with
the same (untyped) reason, which now trips the new unblock-loop breaker
and routes the second block to triage instead of blocked — so only one
'blocked' notification fired. The test's actual intent is that TWO
distinct block cycles each notify; give the two cycles different kinds
(needs_input then capability) so they're genuinely separate blocks. The
same-cause loop→triage path is covered by test_kanban_block_kinds.py.
The kanban-worker and kanban-orchestrator bundled skills existed only to
be force-loaded into dispatcher-spawned workers, gated by
environments:[kanban] so they wouldn't leak into normal CLI listings.
That gating was fragile (the leak that #50443 patched) and the
--skills auto-load was already best-effort — most workers ran without it
because the bundled skill isn't present in profile-scoped skills dirs.
Remove the skills entirely and promote their load-bearing content
(workspace kinds, deliverable artifacts, created-card integrity, profile
discovery) into KANBAN_GUIDANCE, which is already injected into every
kanban worker's system prompt. Net result: every worker reliably gets
the guidance, nothing can leak into a CLI/blank-slate session, and the
gating machinery is gone.
- agent/prompt_builder.py: promote the 4 load-bearing rules into KANBAN_GUIDANCE
- hermes_cli/kanban_db.py: drop --skills kanban-worker auto-injection + _kanban_worker_skill_available probe
- hermes_cli/kanban_swarm.py: drop skills=[kanban-orchestrator] on the root card
- hermes_cli/kanban.py: drop kanban-init skill seeding; fix help text
- delete skills/devops/kanban-{worker,orchestrator}
- docs: delete the two skill pages (EN+zh), fix sidebars/catalog/kanban.md/kanban-worker-lanes.md and the video-orchestrator + codex-lane references
- tests: update spawn-argv expectations; re-bound the guidance-size guard
Supersedes the skill-leak half of #50443 (credit @helix4u for flagging the area).
After a worker crash + reclaim + respawn, the board could show a task in the
Ready lane while its task_run was 'running' and the new worker was actively
executing (#36910). The dispatcher could then treat live work as available and
double-assign.
Root cause: the three reclaim paths (detect_crashed_workers,
release_stale_claims heartbeat-stale backstop, enforce_max_runtime) each
snapshot a task's worker_pid/claim_lock, do liveness work, then reset
tasks.status back to 'ready' with only a 'WHERE status=running' guard. If the
task was reclaimed AND re-claimed by a NEW worker in between (new run, new
claim_lock, live pid), the stale UPDATE clobbered the live task: status flipped
to 'ready' while the fresh run stayed 'running'. claim_task is the only writer
that sets status='running', so nothing put it back — permanent desync.
Fix: gate each reset on the snapshot's claim_lock (and worker_pid where
available) so it only fires when the task is still owned by the worker the
reclaim was computed for. A stale reclaim now no-ops (rowcount 0) instead of
desyncing a re-claimed task. Genuine crashes (lock still matches) reclaim
exactly as before.
This is the same race class the in-gateway dispatch lock (single-writer ticks)
mitigates, closed at the row level so a single dispatcher's fast
reclaim->respawn across two ticks is also safe.
Closes#36910.
connect() wrapped its entire body in an unbounded blocking flock(LOCK_EX) on
every call (_cross_process_init_lock). A single process stalled inside the
critical section — or a stale lock held by a wedged worker — blocked every
other connect(), including the long-lived gateway dispatcher's next-tick
connect, forever. No timeout, no traceback, no recovery: the board silently
stopped being worked until a manual restart (issue #36644).
Two fixes:
1. Fast-path skip: once THIS process has initialized a path, the expensive
first-open work (header validation, integrity probe, schema + additive
migrations) is already cached in _INITIALIZED_PATHS. The steady-state
connect has nothing for the cross-process lock to protect, so it now opens
the connection (WAL + pragmas) under only the cheap in-process _INIT_LOCK
and never touches the file lock. This removes the lock from the dispatcher's
hot path entirely — a stalled external 'hermes kanban list' can no longer
block ticks.
2. Bounded acquire: even on first-init, _cross_process_init_lock now retries a
non-blocking acquire up to a 10s deadline, then logs a WARNING and proceeds
WITHOUT the cross-process lock. Safe because the in-process _INIT_LOCK still
serializes same-process threads and the init work is idempotent
(CREATE TABLE IF NOT EXISTS + additive migrations) — worst case is redundant
work, not corruption. A bounded 'proceed anyway' beats an unbounded hang.
Windows path switched LK_LOCK -> LK_NBLCK (non-blocking) to match.
Closes#36644.
_default_spawn launched the worker subprocess with cwd=workspace and set
HERMES_KANBAN_WORKSPACE, but never set TERMINAL_CWD — so the worker inherited
the dispatching gateway's TERMINAL_CWD. That value takes precedence over the
process cwd in two places:
- tools/file_tools.py::_resolve_base_dir — a relative write_file path resolved
against the gateway user's home instead of the workspace, so artifacts
silently landed outside the workspace (#41312).
- agent_init's context-file loader — AGENTS.md was discovered relative to the
gateway's cwd, so under multi-profile dispatch a worker loaded whichever
gateway won the claim race's AGENTS.md, not the task's (#34619).
Both are the same root cause. Pinning TERMINAL_CWD to the workspace (where the
task's work actually happens) fixes both. Guarded on an existing absolute dir
because file_tools rejects relative/sentinel TERMINAL_CWD values — a non-dir
workspace leaves the inherited value rather than writing a meaningless one.
Closes#34619, closes#41312.
Plugins could observe session/tool/approval lifecycle but had no way to
observe kanban task transitions. Adds three observer hooks fired by the
board's claim/complete/block transitions:
- kanban_task_claimed (dispatcher process, before worker spawn)
- kanban_task_completed (worker process, carries summary)
- kanban_task_blocked (worker process, carries reason)
Each fires AFTER the DB write txn commits, so a plugin observes durable
state and a slow/hanging callback can never hold the SQLite write lock.
All firing is best-effort: a raising hook is logged and swallowed and
never breaks a board transition. profile_name is resolved from
HERMES_HOME so dispatcher- and worker-side hooks carry the right profile.
Requested by @Smithangshu on Discord.
A shell-launched 'hermes gateway run --replace' / 'gateway restart' on a
systemd/launchd host can leave an orphan gateway whose kanban dispatcher
escapes the service cgroup, survives 'systemctl restart', and becomes a
second long-lived writer on the shared kanban.db. Two dispatchers that each
believe they own the file both pass SQLite busy_timeout and then race on WAL
frames — the documented root cause of multi-writer corruption (issue #35240).
The existing _guard_supervised_gateway_conflict startup guard blocks the
common way an orphan is born, but does nothing once a second dispatcher
already exists. This adds the defense-in-depth: dispatch_once now wraps every
tick in a non-blocking, board-scoped flock (_dispatch_tick_lock). A losing
dispatcher returns DispatchResult(skipped_locked=True) and does zero DB writes
this tick — so two dispatchers can never run a reclaim/spawn/write sequence
concurrently regardless of how the second one got there.
- Non-blocking (LOCK_NB): never stalls the gateway's async watcher.
- Board-scoped: lock file is a .dispatch.lock sibling of each board's
kanban.db, so unrelated boards tick in parallel.
- POSIX + Windows (fcntl / msvcrt LK_NBLCK), no-op degrade where neither
exists — mirrors the existing _cross_process_init_lock pattern.
Verified with a real two-process orphan repro: while a separate process holds
the lock, dispatch_once skips; after release it runs.
