mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-12 13:52:15 +00:00
feat(moa): add reference_max_tokens to cap advisor output and cut turn latency (#56756)
MoA per-turn latency is dominated by advisor GENERATION: turn wall time correlates ~0.88 with output tokens and ~-0.03 with input tokens (measured over 52 turns). Each turn waits for the slowest advisor to finish writing, and advisors were uncapped — writing multi-thousand-token essays the aggregator only needs the gist of. Add an opt-in per-preset reference_max_tokens knob (mirrors reference_temperature) that caps ADVISOR output only; the acting aggregator is never capped. Default None = uncapped, so existing presets are byte-for-byte unchanged (no regression). Wired through both MoA execution paths (MoAChatCompletions.create and aggregate_moa_context). E2E: same task, closed preset uncapped vs reference_max_tokens=600 -> 59s to 33s (~44% faster), final answer identical/correct. - hermes_cli/moa_config.py: _coerce_int_or_none helper + reference_max_tokens in _normalize_preset/_default_preset/flattened view - agent/moa_loop.py: read preset.reference_max_tokens, pass to reference fan-out - agent/conversation_loop.py: pass reference_max_tokens on the per-turn path - tests + docs
This commit is contained in:
parent
9be39de0f2
commit
543d305bbb
5 changed files with 117 additions and 5 deletions
|
|
@ -715,10 +715,17 @@ class MoAChatCompletions:
|
|||
# aggregator's spend (often the bulk of the turn) is silently dropped
|
||||
# and the session cost reflects advisor fan-out only.
|
||||
self.last_aggregator_slot = dict(aggregator) if aggregator else None
|
||||
# MoA does not cap reference or aggregator output: each model uses its
|
||||
# own maximum. Passing max_tokens=None makes call_llm omit the parameter
|
||||
# (it never caps by default), so a long aggregator synthesis is never
|
||||
# truncated and providers that reject max_tokens don't 400.
|
||||
# By default MoA does not cap reference or aggregator output: each model
|
||||
# uses its own maximum (max_tokens=None → call_llm omits the parameter,
|
||||
# so a long aggregator synthesis is never truncated and providers that
|
||||
# reject max_tokens don't 400). A preset MAY set reference_max_tokens to
|
||||
# cap ADVISOR output only — advisor generation is the dominant MoA
|
||||
# latency (turn latency correlates ~0.88 with output tokens), and the
|
||||
# aggregator only needs the gist of each advisor's judgement, so a cap
|
||||
# (e.g. 600) measurably cuts per-turn wall time (~44% on a sample task).
|
||||
# The acting aggregator is never capped here (its output is the
|
||||
# user-visible answer).
|
||||
reference_max_tokens = preset.get("reference_max_tokens")
|
||||
temperature = float(preset.get("reference_temperature", 0.6) or 0.6)
|
||||
aggregator_temperature = float(preset.get("aggregator_temperature", api_kwargs.get("temperature") or 0.4) or 0.4)
|
||||
|
||||
|
|
@ -762,7 +769,7 @@ class MoAChatCompletions:
|
|||
reference_models,
|
||||
ref_messages,
|
||||
temperature=temperature,
|
||||
max_tokens=None,
|
||||
max_tokens=reference_max_tokens,
|
||||
)
|
||||
self._ref_cache_key = _cache_key
|
||||
self._ref_cache_outputs = list(reference_outputs)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue