feat(goals): /goal checklist + /subgoal user controls (#23456)

* feat(goals): /goal checklist + /subgoal user controls

Two-phase judge for /goal — Phase A decomposes the goal into a detailed
checklist on first turn; Phase B evaluates each pending item harshly
against the agent's most recent response. The goal completes only when
every item is in a terminal status (completed or impossible). Adds
/subgoal so the user can append, complete, mark impossible, undo,
remove, or clear items the judge missed or got wrong.

Mechanics:
- GoalState gains `checklist` and `decomposed` fields, both backwards
  compatible (old state_meta rows load unchanged).
- Phase A: aux call writes a harsh, exhaustive checklist; biased toward
  more items not fewer. Falls through to legacy freeform judge when
  decompose fails.
- Phase B: judge gets the checklist + last-response snippet + path to
  a per-session conversation dump at <HERMES_HOME>/goals/<sid>.json.
  A bounded read_file tool (max 5 calls per turn, restricted to that
  one file) lets the judge inspect history when the snippet is
  ambiguous. Stickiness in code: terminal items are frozen, only the
  user can revert via /subgoal undo.
- Continuation prompt shows checklist progress when non-empty;
  reverts to old prompt when empty.
- Status line shows M/N done counts.

CLI + gateway + TUI gateway all pass the agent reference into
evaluate_after_turn so the dump can be written. Gateway-side
/subgoal is allowed mid-run since it only modifies the checklist
the judge consults at turn boundaries.

Tests: 24 new cases — backcompat round-trip, Phase A decompose,
Phase B updates + new_items + stickiness, user override flows,
conversation dump (incl. unsafe-sid sanitization), judge read_file
restriction. Existing freeform-mode tests updated to patch the
renamed `judge_goal_freeform` and skip Phase A explicitly.

* fix(goals): off-by-one in judge index, message-list plumbing, prompt tuning

Three live-test findings from running /goal end-to-end against
gemini-3-flash-preview as the judge:

1. Off-by-one bug — the judge sees the checklist rendered with 1-based
   indices ('1. [ ] foo, 2. [ ] bar') but the apply layer indexed
   state.checklist as 0-based. Result: every judge update landed on
   the wrong item, evidence got attached to neighbouring rows, and
   the genuine 'first pending' item (usually #1) never got marked.
   Fix: convert 1 → 0 in _parse_evaluate_response. Also tightened the
   user prompt to call out the 1-based scheme explicitly. New tests
   cover the parser conversion + an end-to-end fake-judge round-trip.

2. Conversation dump never happened — _extract_agent_messages tried
   common AIAgent attribute names (.messages, .conversation_history,
   etc.) but AIAgent doesn't expose the message list as an instance
   attribute; it lives inside run_conversation()'s scope. Result: the
   judge's read_file tool always saw history_path=unavailable. Fix:
   added an explicit messages= kwarg to evaluate_after_turn that all
   three call sites (CLI, gateway, TUI gateway) now pass directly.
   Agent-attribute extraction kept as back-compat fallback.

3. Prompt was too harsh on simple goals. The original 'be HARSH,
   default to leaving items pending' wording made the judge refuse
   to mark 'file exists' completed even after the agent ran ls,
   test -f, os.path.isfile, and find — burning the entire 8-turn
   budget on a fizzbuzz task. Softened to 'strict but not absurd'
   with explicit guidance on what counts as evidence and a directive
   not to require re-proving items already established earlier.

Re-tested live with the same fizzbuzz goal: now terminates in 2
turns with all 8 checklist items correctly attributed to their
own evidence. /subgoal user-action flow (add / complete / undo /
impossible) verified live as well.
This commit is contained in:
Teknium 2026-05-10 16:56:51 -07:00 committed by GitHub
parent c0bbdec850
commit 404640a2b7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 1860 additions and 113 deletions

View file

@ -58,6 +58,11 @@ def _make_cli_with_goal(session_id: str, goal_text: str = "build a thing"):
mgr = GoalManager(session_id=session_id, default_max_turns=5)
mgr.set(goal_text)
# Skip Phase-A decompose so tests can patch judge_goal_freeform directly
# for legacy verdict assertions.
mgr.state.decomposed = True
from hermes_cli.goals import save_goal as _sg
_sg(mgr.session_id, mgr.state)
cli._goal_manager = mgr
return cli, mgr
@ -81,7 +86,7 @@ class TestInterruptAutoPause:
# Judge MUST NOT run on an interrupted turn. If it does, we've
# regressed — fail loudly instead of silently querying a mock.
with patch("hermes_cli.goals.judge_goal") as judge_mock:
with patch("hermes_cli.goals.judge_goal_freeform") as judge_mock:
judge_mock.side_effect = AssertionError(
"judge_goal called on an interrupted turn"
)
@ -106,7 +111,7 @@ class TestInterruptAutoPause:
cli.conversation_history = [
{"role": "assistant", "content": "partial"},
]
with patch("hermes_cli.goals.judge_goal"):
with patch("hermes_cli.goals.judge_goal_freeform"):
cli._maybe_continue_goal_after_turn()
assert mgr.state.status == "paused"
@ -125,7 +130,7 @@ class TestEmptyResponseSkip:
{"role": "assistant", "content": " \n\n "},
]
with patch("hermes_cli.goals.judge_goal") as judge_mock:
with patch("hermes_cli.goals.judge_goal_freeform") as judge_mock:
judge_mock.side_effect = AssertionError(
"judge_goal called on an empty response"
)
@ -144,7 +149,7 @@ class TestEmptyResponseSkip:
{"role": "user", "content": "go"},
]
with patch("hermes_cli.goals.judge_goal") as judge_mock:
with patch("hermes_cli.goals.judge_goal_freeform") as judge_mock:
judge_mock.side_effect = AssertionError(
"judge_goal called without an assistant response"
)
@ -169,7 +174,7 @@ class TestHealthyTurnStillRuns:
# Force the judge to say "continue" without touching the network.
with patch(
"hermes_cli.goals.judge_goal",
"hermes_cli.goals.judge_goal_freeform",
return_value=("continue", "needs more steps", False),
):
cli._maybe_continue_goal_after_turn()
@ -189,7 +194,7 @@ class TestHealthyTurnStillRuns:
]
with patch(
"hermes_cli.goals.judge_goal",
"hermes_cli.goals.judge_goal_freeform",
return_value=("done", "goal satisfied", False),
):
cli._maybe_continue_goal_after_turn()