feat(moa): add reference_max_tokens to cap advisor output and cut turn latency (#56756)

MoA per-turn latency is dominated by advisor GENERATION: turn wall time
correlates ~0.88 with output tokens and ~-0.03 with input tokens (measured over
52 turns). Each turn waits for the slowest advisor to finish writing, and
advisors were uncapped — writing multi-thousand-token essays the aggregator
only needs the gist of.

Add an opt-in per-preset reference_max_tokens knob (mirrors reference_temperature)
that caps ADVISOR output only; the acting aggregator is never capped. Default
None = uncapped, so existing presets are byte-for-byte unchanged (no regression).
Wired through both MoA execution paths (MoAChatCompletions.create and
aggregate_moa_context).

E2E: same task, closed preset uncapped vs reference_max_tokens=600 -> 59s to 33s
(~44% faster), final answer identical/correct.

- hermes_cli/moa_config.py: _coerce_int_or_none helper + reference_max_tokens
  in _normalize_preset/_default_preset/flattened view
- agent/moa_loop.py: read preset.reference_max_tokens, pass to reference fan-out
- agent/conversation_loop.py: pass reference_max_tokens on the per-turn path
- tests + docs
This commit is contained in:
Teknium 2026-07-02 00:16:35 -07:00 committed by GitHub
parent 9be39de0f2
commit 543d305bbb
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 117 additions and 5 deletions

View file

@ -715,10 +715,17 @@ class MoAChatCompletions:
# aggregator's spend (often the bulk of the turn) is silently dropped
# and the session cost reflects advisor fan-out only.
self.last_aggregator_slot = dict(aggregator) if aggregator else None
# MoA does not cap reference or aggregator output: each model uses its
# own maximum. Passing max_tokens=None makes call_llm omit the parameter
# (it never caps by default), so a long aggregator synthesis is never
# truncated and providers that reject max_tokens don't 400.
# By default MoA does not cap reference or aggregator output: each model
# uses its own maximum (max_tokens=None → call_llm omits the parameter,
# so a long aggregator synthesis is never truncated and providers that
# reject max_tokens don't 400). A preset MAY set reference_max_tokens to
# cap ADVISOR output only — advisor generation is the dominant MoA
# latency (turn latency correlates ~0.88 with output tokens), and the
# aggregator only needs the gist of each advisor's judgement, so a cap
# (e.g. 600) measurably cuts per-turn wall time (~44% on a sample task).
# The acting aggregator is never capped here (its output is the
# user-visible answer).
reference_max_tokens = preset.get("reference_max_tokens")
temperature = float(preset.get("reference_temperature", 0.6) or 0.6)
aggregator_temperature = float(preset.get("aggregator_temperature", api_kwargs.get("temperature") or 0.4) or 0.4)
@ -762,7 +769,7 @@ class MoAChatCompletions:
reference_models,
ref_messages,
temperature=temperature,
max_tokens=None,
max_tokens=reference_max_tokens,
)
self._ref_cache_key = _cache_key
self._ref_cache_outputs = list(reference_outputs)