feat(moa): default advisor fanout to user_turn — the cheapest cadence

Flips the default fan-out cadence from per_iteration (advisors re-run on
every tool iteration, multiplying advisor spend by tool-loop depth) to
user_turn (advisors run once on the first message of each user turn; the
acting aggregator works the rest of the tool loop with that turn's
advice). Until per-mode benchmarks justify a costlier default, MoA
defaults to the cheapest, lowest-impact cadence (#67199).

One default for everyone — no split legacy/new-preset semantics; presets
that want per-step advising set fanout: per_iteration explicitly. All
three modes (user_turn / per_iteration / every_n:N) remain selectable;
every_n:1 still collapses to per_iteration (semantic identity), while
unparseable values now fall to user_turn (the default).

Docs updated with a default-change note; the per-iteration rerun test
pins its mode explicitly.

Co-authored-by: skyer-flyyy <188930297+skyer-flyyy@users.noreply.github.com>
This commit is contained in:
Teknium 2026-07-23 20:11:57 -07:00
parent b0ef72a8a0
commit 23476207bc
7 changed files with 84 additions and 56 deletions

View file

@ -1703,14 +1703,16 @@ class MoAChatCompletions:
reference_outputs: list[tuple[str, str, Any]] = []
ref_messages = _reference_messages(messages)
# Fan-out cadence. "per_iteration" (default): advisors re-run whenever
# the advisory view changes — i.e. every tool iteration, since the
# view grows with each tool result. "user_turn": advisors run ONCE per
# user turn; subsequent tool iterations reuse that turn's advice and
# the aggregator acts alone (the original MoA shape: synthesize at the
# start, then let the acting model work). Implemented by hashing only
# the prefix up to the LAST USER message so mid-turn growth doesn't
# change the signature — iteration 2+ becomes a cache HIT.
# Fan-out cadence. "user_turn" (default — cheapest cadence, #67199):
# advisors run ONCE per user turn; subsequent tool iterations reuse
# that turn's advice and the aggregator acts alone (the original MoA
# shape: synthesize at the start, then let the acting model work).
# Implemented by hashing only the prefix up to the LAST USER message
# so mid-turn growth doesn't change the signature — iteration 2+
# becomes a cache HIT. "per_iteration": advisors re-run whenever the
# advisory view changes — i.e. every tool iteration, since the view
# grows with each tool result; advice tracks live task state at the
# cost of multiplying advisor latency/spend by tool-loop depth.
# "every_n:<N>" (N >= 2): the middle ground (issue #63393 — advisor
# fan-out multiplies latency/cost by the tool-iteration count).
# Advisors run on iteration 1 of a user turn and then every Nth tool
@ -1720,7 +1722,7 @@ class MoAChatCompletions:
# refreshed against the very latest tool results). The iteration
# counter is scoped per user turn and resets on a new user message,
# so every turn starts with fresh advice.
fanout_mode = str(preset.get("fanout") or "per_iteration").strip().lower()
fanout_mode = str(preset.get("fanout") or "user_turn").strip().lower()
every_n = 0
if fanout_mode.startswith("every_n:"):
try:
@ -1728,8 +1730,8 @@ class MoAChatCompletions:
except (TypeError, ValueError):
every_n = 0
if every_n < 2:
# Unparseable / degenerate cadence degrades to the default,
# mirroring _coerce_fanout's tolerant-read contract.
# every_n:1 semantically IS per-iteration; degrade there,
# mirroring _coerce_fanout's collapse of degenerate N.
fanout_mode = "per_iteration"
sig_messages = ref_messages
turn_prefix = ref_messages

View file

@ -1017,7 +1017,7 @@ export interface MoaConfigResponse {
reference_temperature: number
/** Optional advisor output cap — round-tripped, not edited here. */
reference_max_tokens?: number | null
/** Fan-out cadence (per_iteration | user_turn) — round-tripped. */
/** Fan-out cadence (user_turn default | per_iteration | every_n:N) — round-tripped. */
fanout?: string
reference_timeout: number | null
}

View file

@ -108,9 +108,10 @@ def _coerce_fanout(value: Any) -> str:
``every_n:<N>`` (N >= 2). The ``every_n`` cadence also accepts the mapping
form ``{mode: every_n, n: N}`` from hand-edited YAML and normalizes it to
the canonical string, so the rest of the pipeline (presets, flattened
view, runtime) only ever sees one shape. ``every_n:1`` means "run every
iteration" and collapses to ``per_iteration``; anything unparseable falls
back to ``per_iteration`` (the tolerant-read contract of this module).
view, runtime) only ever sees one shape. ``every_n:1`` semantically means
"run every iteration" and collapses to ``per_iteration``; anything
unparseable falls back to ``user_turn`` (the default cheapest cadence;
see #67199).
"""
if isinstance(value, dict):
# Mapping form: {mode: every_n, n: 3}. Non-every_n mapping modes fall
@ -118,7 +119,9 @@ def _coerce_fanout(value: Any) -> str:
mode = str(value.get("mode") or "").strip().lower()
if mode == "every_n":
n = _coerce_int(value.get("n"), 0)
return f"every_n:{n}" if n >= 2 else "per_iteration"
if n >= 2:
return f"every_n:{n}"
return "per_iteration" if n == 1 else "user_turn"
value = mode
mode = str(value or "").strip().lower()
if mode in {"per_iteration", "user_turn"}:
@ -128,7 +131,9 @@ def _coerce_fanout(value: Any) -> str:
n = _coerce_int(rest.strip(), 0) if sep else 0
if n >= 2:
return f"every_n:{n}"
return "per_iteration"
if n == 1:
return "per_iteration"
return "user_turn"
def coerce_privacy_filter(value: Any) -> str:
@ -300,7 +305,7 @@ def _default_preset() -> dict[str, Any]:
"degraded_reference_policy": "loud",
"max_tokens": 4096,
"reference_max_tokens": None,
"fanout": "per_iteration",
"fanout": "user_turn",
"enabled": True,
}
@ -348,16 +353,18 @@ def _normalize_preset(raw: Any) -> dict[str, Any]:
# judgement, so capping roughly halves per-turn wall time. Does NOT cap
# the acting aggregator (its output is the user-visible answer).
"reference_max_tokens": _coerce_int_or_none(raw.get("reference_max_tokens")),
# When the reference fan-out runs. "per_iteration" (default) re-runs
# the advisors whenever the advisory view changes — i.e. every tool
# iteration, so advice tracks live task state. "user_turn" runs the
# advisors ONCE per user turn (the original MoA shape): the
# aggregator gets their upfront plan-level advice, then acts alone
# for the rest of the tool loop. "every_n:<N>" (N >= 2) is the middle
# ground: advisors run on the first iteration of each user turn and
# every Nth tool iteration after it; in-between iterations reuse the
# cached guidance from the last advisor run. Also accepts the mapping
# form {mode: every_n, n: N}, normalized to the canonical string.
# When the reference fan-out runs. "user_turn" (default) runs the
# advisors ONCE per user turn (the original MoA shape, and the
# cheapest cadence — #67199): the aggregator gets their upfront
# plan-level advice, then acts alone for the rest of the tool loop.
# "per_iteration" re-runs the advisors whenever the advisory view
# changes — i.e. every tool iteration, so advice tracks live task
# state at the cost of multiplying advisor spend by tool-loop depth.
# "every_n:<N>" (N >= 2) is the middle ground: advisors run on the
# first iteration of each user turn and every Nth tool iteration
# after it; in-between iterations reuse the cached guidance from the
# last advisor run. Also accepts the mapping form
# {mode: every_n, n: N}, normalized to the canonical string.
"fanout": _coerce_fanout(raw.get("fanout")),
}
@ -407,7 +414,7 @@ def normalize_moa_config(raw: Any) -> dict[str, Any]:
"degraded_reference_policy": active["degraded_reference_policy"],
"max_tokens": active["max_tokens"],
"reference_max_tokens": active.get("reference_max_tokens"),
"fanout": active.get("fanout", "per_iteration"),
"fanout": active.get("fanout", "user_turn"),
"enabled": active["enabled"],
# MoA-level (not per-preset) toggles ride at the top level alongside
# save_traces. privacy_filter: '' (off, default) | 'display' | 'full'

View file

@ -672,8 +672,14 @@ def test_slot_max_tokens_absent_by_default():
# --- fanout cadence normalization (every_n) ---
def test_fanout_defaults_to_per_iteration():
def test_fanout_defaults_to_user_turn():
# Default is the cheapest cadence (#67199): advisors once per user turn.
cfg = normalize_moa_config({})
assert cfg["fanout"] == "user_turn"
def test_fanout_per_iteration_still_selectable():
cfg = normalize_moa_config({"fanout": "per_iteration"})
assert cfg["fanout"] == "per_iteration"
@ -689,14 +695,15 @@ def test_fanout_every_n_mapping_form_normalized_to_string():
def test_fanout_every_n_degenerate_n_falls_back():
# n=1 means "every iteration" — that IS per_iteration; n=0 / negative /
# garbage must never produce a broken cadence string.
# n=1 means "every iteration" — that semantically IS per_iteration;
# n=0 / negative / garbage is unparseable and falls to the default
# cadence (user_turn, the cheapest — #67199).
assert normalize_moa_config({"fanout": "every_n:1"})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": "every_n:0"})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": "every_n:-2"})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": "every_n:x"})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": "every_n"})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": {"mode": "every_n"}})["fanout"] == "per_iteration"
assert normalize_moa_config({"fanout": "every_n:0"})["fanout"] == "user_turn"
assert normalize_moa_config({"fanout": "every_n:-2"})["fanout"] == "user_turn"
assert normalize_moa_config({"fanout": "every_n:x"})["fanout"] == "user_turn"
assert normalize_moa_config({"fanout": "every_n"})["fanout"] == "user_turn"
assert normalize_moa_config({"fanout": {"mode": "every_n"}})["fanout"] == "user_turn"
def test_fanout_every_n_round_trips_through_normalize():

View file

@ -1177,10 +1177,11 @@ def test_references_parallel_interrupt_aborts_wait(monkeypatch):
release_wedged.set() # don't leak a blocked thread past the test
def _ref_config(home):
def _ref_config(home, fanout: str | None = None):
home.mkdir()
fanout_line = f"\n fanout: {fanout}" if fanout else ""
(home / "config.yaml").write_text(
"""
f"""
moa:
default_preset: review
presets:
@ -1192,7 +1193,7 @@ moa:
model: anthropic/claude-opus-4.8
aggregator:
provider: openrouter
model: anthropic/claude-opus-4.8
model: anthropic/claude-opus-4.8{fanout_line}
""".strip(),
encoding="utf-8",
)
@ -1234,13 +1235,15 @@ def test_moa_facade_emits_reference_then_aggregating(monkeypatch, tmp_path):
def test_moa_facade_reruns_references_on_new_tool_result(monkeypatch, tmp_path):
"""References re-run when a new tool result advances the task state.
The agent loop calls create() once per tool-loop iteration. References must
judge the LATEST state, so a new tool result is a cache MISS and re-runs the
references but a redundant create() call with the SAME state is a cache
HIT (no re-run, no re-emit), so we don't fire on a pure no-op re-call.
Pins fanout: per_iteration explicitly (the default became user_turn,
#67199). In this mode the agent loop calls create() once per tool-loop
iteration and references must judge the LATEST state, so a new tool
result is a cache MISS and re-runs the references but a redundant
create() call with the SAME state is a cache HIT (no re-run, no
re-emit), so we don't fire on a pure no-op re-call.
"""
home = tmp_path / ".hermes"
_ref_config(home)
_ref_config(home, fanout="per_iteration")
monkeypatch.setenv("HERMES_HOME", str(home))
ref_runs = []

View file

@ -2349,7 +2349,7 @@ export interface MoaConfigResponse {
max_tokens: number;
/** Optional advisor output cap — round-tripped, not edited here. */
reference_max_tokens?: number | null;
/** Fan-out cadence (per_iteration | user_turn) — round-tripped. */
/** Fan-out cadence (user_turn default | per_iteration | every_n:N) — round-tripped. */
fanout?: string;
enabled: boolean;
}>;

View file

@ -134,14 +134,16 @@ Leave it unset (or `0`/blank) to keep the prior uncapped behavior.
### Advisor cadence with `fanout`
By default the advisors re-run on **every tool iteration** (`fanout:
per_iteration`), so their advice always tracks the latest tool results — at
the cost of multiplying advisor latency and spend by the number of tool calls
in a turn. Two alternative cadences trade freshness for speed:
By default the advisors run **once per user turn** (`fanout: user_turn`) —
they synthesize plan-level advice on the first message of the turn, then the
acting aggregator works through the rest of the tool loop alone. This is the
cheapest cadence: advisor cost does not multiply with the number of tool
calls in a turn. Two alternative cadences trade cost for advice freshness:
- `fanout: user_turn` — advisors run **once per user turn**; every tool
iteration after that reuses the same upfront advice (the original MoA
shape: synthesize a plan, then let the acting model work).
- `fanout: per_iteration` — advisors re-run on **every tool iteration**, so
their advice always tracks the latest tool results — at the cost of
multiplying advisor latency and spend by the number of tool calls in a
turn.
- `fanout: every_n:3` — the middle ground: advisors run on the **first**
iteration of each user turn and then every **3rd** tool iteration (any
`N >= 2` works). Iterations in between reuse the cached guidance from the
@ -154,17 +156,24 @@ in a turn. Two alternative cadences trade freshness for speed:
```yaml
moa:
presets:
fast:
fresh:
reference_models:
- provider: openrouter
model: anthropic/claude-opus-4.8
aggregator:
provider: openrouter
model: openai/gpt-5.5
fanout: every_n:3 # advisors refresh every 3rd tool iteration
fanout: per_iteration # advisors refresh on every tool iteration
```
Unknown or malformed values fall back to `per_iteration`.
Unknown or malformed values fall back to `user_turn`.
:::note Default change
Prior to July 2026 the default cadence was `per_iteration`. The default is
now `user_turn` — the cheapest, lowest-impact cadence — until per-mode
benchmarks justify a costlier default. Presets that want per-step advising
back set `fanout: per_iteration` explicitly.
:::
### Privacy filter for advisor outputs