From 3cd8feb63c5c17175bfc635b542c6123c6d420b8 Mon Sep 17 00:00:00 2001 From: slow4cyl <22582211+slow4cyl@users.noreply.github.com> Date: Sat, 11 Jul 2026 17:53:34 -0500 Subject: [PATCH] docs(kanban): worker-lifecycle + events table reflect the bounded protocol-violation retry Fold in kevinb361's suggested lifecycle wording (#61817 conceded in favor of this PR) and update the second stale site his sweep didn't cover: the task-events table still said the dispatcher 'auto-blocks immediately instead of retrying'. Both now describe the violation-only streak: protocol_violation fires on every violation (its payload marker feeds the budget), below-budget runs return the task to ready, and gave_up + auto-block happen only when the consecutive streak reaches _PROTOCOL_VIOLATION_FAILURE_LIMIT (default 3, per-task max_retries overriding). --- website/docs/user-guide/features/kanban.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/website/docs/user-guide/features/kanban.md b/website/docs/user-guide/features/kanban.md index b505ccc9bf4a..57d66f585180 100644 --- a/website/docs/user-guide/features/kanban.md +++ b/website/docs/user-guide/features/kanban.md @@ -369,10 +369,17 @@ Every profile that works kanban tasks automatically gets the worker lifecycle That final `kanban_complete` / `kanban_block` call is part of the worker protocol. If the worker process exits with status 0 while the task is still -`running`, the dispatcher treats that as a protocol violation, emits a -`protocol_violation` event, and auto-blocks the task on the next tick instead -of respawning it into the same loop. This usually means the model wrote a -plain-text answer and exited without using the Kanban tool surface. +`running`, the dispatcher treats that as a protocol violation and emits a +`protocol_violation` event. Because a protocol violation is **not +deterministic** — the model may emit the missing tool call on a later run — +the dispatcher gives it a **bounded retry** (up to +`_PROTOCOL_VIOLATION_FAILURE_LIMIT` consecutive violations, default 3) before +auto-blocking the task instead of respawning it into the same loop. The budget +counts only *consecutive* clean-exit protocol violations — interleaved +rate-limited requeues are neutral, and any other failure kind resets the +streak — and a per-task `max_retries` overrides the bound. This usually means +the model wrote a plain-text answer and exited without using the Kanban tool +surface. The lifecycle plus the load-bearing reference details (workspace kinds, deliverable `artifacts`, claiming created cards) ship in that system-prompt block, so every worker has them regardless of which profile it runs under — no per-profile skill setup required. @@ -926,7 +933,7 @@ Every transition appends a row to `task_events`. Each row carries an optional `r | `stale` | `{elapsed_seconds, last_heartbeat_at, heartbeat_age_seconds, timeout_seconds, pid, terminated}` | Task ran longer than `kanban.dispatch_stale_timeout_seconds` (default 4 h) AND no `kanban_heartbeat` arrived in the last hour. Dispatcher SIGTERM'd the host-local worker (if any), reset the task to `ready` for re-dispatch. Does NOT tick the failure counter (stale is dispatcher-side absence detection, not a worker fault). Workers running long operations should call `kanban_heartbeat` at least once an hour to avoid this. | | `respawn_guarded` | `{reason}` | Dispatcher refused to re-spawn this ready task this tick. Reasons: `blocker_auth` (last failure was a quota/auth/429 error — wait for the rate window to reset), `recent_success` (a completed run happened in the last hour — wait for review before re-running), `active_pr` (a GitHub PR URL appears in a recent comment — a prior worker already opened a PR). The task stays in `ready`; the next tick gets another chance to spawn. If the underlying condition persists, the normal `consecutive_failures` circuit breaker will auto-block via `gave_up` after `failure_limit` failures. | | `spawn_failed` | `{error, failures}` | One spawn attempt failed (missing PATH, workspace unmountable, …). Counter increments; task returns to `ready` for retry. | -| `protocol_violation` | `{pid, claimer, exit_code}` | Worker exited successfully while the task was still `running`, usually because it answered without calling `kanban_complete` or `kanban_block`. The dispatcher also emits `gave_up` and auto-blocks immediately instead of retrying. | +| `protocol_violation` | `{pid, claimer, exit_code, protocol_violation}` | Worker exited successfully while the task was still `running`, usually because it answered without calling `kanban_complete` or `kanban_block`. Emitted on every violation (the payload's `protocol_violation: true` marker is copied into the run metadata and feeds the violation-only retry budget). Below the budget — up to `_PROTOCOL_VIOLATION_FAILURE_LIMIT` (default 3) *consecutive* violations, per-task `max_retries` overriding — the task simply returns to `ready` for another attempt; when the streak reaches the bound the dispatcher also emits `gave_up` and auto-blocks. | | `gave_up` | `{failures, effective_limit, limit_source, error}` | Circuit breaker fired after N consecutive non-successful attempts. Task auto-blocks with the last error. The effective limit resolves as task `max_retries`, then dispatcher `failure_limit` / `kanban.failure_limit`, then the built-in default. | `hermes kanban tail ` shows these for a single task. `hermes kanban watch` streams them board-wide.