mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-20 15:33:54 +00:00
follow-up: integrate agent nudge + dispatcher retry docs and tests
- Nudge text now warns that repeated protocol violations will block the task and require manual intervention, so the model understands the consequence of ignoring the nudge. - Kanban docs restructured to clearly separate the two defense layers: agent-side prevention (nudge, from #64350) and dispatcher-side recovery (bounded retry, from this PR). - Two new integration tests verifying the nudge mentions blocking and that the agent-side and dispatcher-side budgets are independent.
This commit is contained in:
parent
3cd8feb63c
commit
52cafa6f8e
3 changed files with 55 additions and 7 deletions
|
|
@ -96,7 +96,8 @@ def build_kanban_stop_nudge(
|
|||
"1. Finish any remaining deliverable (write the required file(s) now).\n"
|
||||
"2. Call `kanban_complete(summary=..., artifacts=[...])` if the work "
|
||||
"is done, OR `kanban_block(reason=...)` if you are blocked.\n\n"
|
||||
"Never end a turn with only a promise of future action.]"
|
||||
"Never end a turn with only a promise of future action. Repeated "
|
||||
"protocol violations will block this task and require manual intervention.]"
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -93,3 +93,41 @@ def test_nudge_budget_exhausted(clear_kanban_env):
|
|||
assert build_kanban_stop_nudge(messages=[], attempts=2) is None
|
||||
assert build_kanban_stop_nudge(messages=[], attempts=1, max_attempts=1) is None
|
||||
assert build_kanban_stop_nudge(messages=[], attempts=0, max_attempts=1) is not None
|
||||
|
||||
|
||||
# ── Integration: agent nudge + dispatcher bounded retry ──────────────
|
||||
# These tests verify the two layers compose correctly: the agent-side
|
||||
# nudge fires first (up to 2 attempts), and if the worker still exits
|
||||
# without a terminal call, the dispatcher's bounded retry (streak of 3)
|
||||
# handles it. See also tests/hermes_cli/test_kanban_core_functionality.py
|
||||
# for the dispatcher-side streak tests.
|
||||
|
||||
|
||||
def test_nudge_text_warns_about_blocking(clear_kanban_env):
|
||||
"""The nudge should warn that repeated violations will block the task."""
|
||||
clear_kanban_env.setenv("HERMES_KANBAN_TASK", "t_abc")
|
||||
nudge = build_kanban_stop_nudge(messages=[], attempts=0)
|
||||
assert nudge is not None
|
||||
assert "block" in nudge.lower(), (
|
||||
"nudge should warn that repeated violations will block the task"
|
||||
)
|
||||
|
||||
|
||||
def test_nudge_and_dispatcher_budgets_are_independent(clear_kanban_env):
|
||||
"""Agent-side nudge budget (2) and dispatcher-side streak (3) are
|
||||
separate budgets — the nudge counter does not affect the dispatcher's
|
||||
violation streak, and vice versa.
|
||||
|
||||
This is a source-level invariant check: the nudge counter
|
||||
(``_kanban_stop_nudges``) lives on the AIAgent instance and resets
|
||||
per session, while the dispatcher streak lives in the task_runs DB
|
||||
table and persists across worker respawns.
|
||||
"""
|
||||
clear_kanban_env.setenv("HERMES_KANBAN_TASK", "t_abc")
|
||||
# Agent-side: 2 nudge attempts per session
|
||||
assert build_kanban_stop_nudge(messages=[], attempts=0) is not None
|
||||
assert build_kanban_stop_nudge(messages=[], attempts=1) is not None
|
||||
assert build_kanban_stop_nudge(messages=[], attempts=2) is None
|
||||
# Dispatcher-side streak is tracked in the DB, not in the nudge module —
|
||||
# the nudge module has no knowledge of the streak counter.
|
||||
assert not hasattr(build_kanban_stop_nudge, "_streak")
|
||||
|
|
|
|||
|
|
@ -370,12 +370,21 @@ Every profile that works kanban tasks automatically gets the worker lifecycle
|
|||
That final `kanban_complete` / `kanban_block` call is part of the worker
|
||||
protocol. If the worker process exits with status 0 while the task is still
|
||||
`running`, the dispatcher treats that as a protocol violation and emits a
|
||||
`protocol_violation` event. Because a protocol violation is **not
|
||||
deterministic** — the model may emit the missing tool call on a later run —
|
||||
the dispatcher gives it a **bounded retry** (up to
|
||||
`_PROTOCOL_VIOLATION_FAILURE_LIMIT` consecutive violations, default 3) before
|
||||
auto-blocking the task instead of respawning it into the same loop. The budget
|
||||
counts only *consecutive* clean-exit protocol violations — interleaved
|
||||
`protocol_violation` event.
|
||||
|
||||
**Agent-side prevention:** Before the worker exits, Hermes injects up to two
|
||||
synthetic nudges when it detects the model is about to stop without a terminal
|
||||
board tool call. This catches the common case where the model narrates the next
|
||||
step ("Let me write the report") and stops with `finish_reason=stop`. The nudge
|
||||
reminds the model to call `kanban_complete` or `kanban_block` immediately. This
|
||||
guard is active only for dispatcher-spawned workers (`HERMES_KANBAN_TASK` is
|
||||
set) and can be disabled with `HERMES_KANBAN_STOP_NUDGE=0`.
|
||||
|
||||
**Dispatcher-side recovery:** If the nudges are exhausted or the worker crashes
|
||||
before reaching the nudge, the dispatcher gives the violation a **bounded retry**
|
||||
(up to `_PROTOCOL_VIOLATION_FAILURE_LIMIT` consecutive violations, default 3)
|
||||
before auto-blocking the task instead of respawning it into the same loop. The
|
||||
budget counts only *consecutive* clean-exit protocol violations — interleaved
|
||||
rate-limited requeues are neutral, and any other failure kind resets the
|
||||
streak — and a per-task `max_retries` overrides the bound. This usually means
|
||||
the model wrote a plain-text answer and exited without using the Kanban tool
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue