mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-08 13:12:08 +00:00
feat(moa): add reference_max_tokens to cap advisor output and cut turn latency (#56756)
MoA per-turn latency is dominated by advisor GENERATION: turn wall time correlates ~0.88 with output tokens and ~-0.03 with input tokens (measured over 52 turns). Each turn waits for the slowest advisor to finish writing, and advisors were uncapped — writing multi-thousand-token essays the aggregator only needs the gist of. Add an opt-in per-preset reference_max_tokens knob (mirrors reference_temperature) that caps ADVISOR output only; the acting aggregator is never capped. Default None = uncapped, so existing presets are byte-for-byte unchanged (no regression). Wired through both MoA execution paths (MoAChatCompletions.create and aggregate_moa_context). E2E: same task, closed preset uncapped vs reference_max_tokens=600 -> 59s to 33s (~44% faster), final answer identical/correct. - hermes_cli/moa_config.py: _coerce_int_or_none helper + reference_max_tokens in _normalize_preset/_default_preset/flattened view - agent/moa_loop.py: read preset.reference_max_tokens, pass to reference fan-out - agent/conversation_loop.py: pass reference_max_tokens on the per-turn path - tests + docs
This commit is contained in:
parent
9be39de0f2
commit
543d305bbb
5 changed files with 117 additions and 5 deletions
|
|
@ -97,6 +97,38 @@ Default preset:
|
|||
- reference: `openrouter:deepseek/deepseek-v4-pro`
|
||||
- aggregator / acting model: `openrouter:anthropic/claude-opus-4.8`
|
||||
|
||||
### Tuning advisor speed with `reference_max_tokens`
|
||||
|
||||
Each turn, MoA runs the reference models (advisors) in parallel and then the
|
||||
aggregator acts. Advisor generation is the dominant per-turn latency — turn
|
||||
wall time correlates strongly with how many tokens the advisors emit, because
|
||||
the turn waits for the slowest advisor to finish writing. By default advisors
|
||||
are **uncapped** (`reference_max_tokens` unset), so they may write long,
|
||||
essay-length advice.
|
||||
|
||||
Set `reference_max_tokens` on a preset to cap advisor output and give concise
|
||||
advice instead. The aggregator only needs the gist of each advisor's
|
||||
judgement, so a cap (e.g. `600`) measurably cuts per-turn wall time with little
|
||||
quality impact. It caps **advisors only** — the acting aggregator's output (the
|
||||
user-visible answer) is never capped.
|
||||
|
||||
```yaml
|
||||
moa:
|
||||
presets:
|
||||
fast:
|
||||
reference_models:
|
||||
- provider: openrouter
|
||||
model: anthropic/claude-opus-4.8
|
||||
- provider: openrouter
|
||||
model: openai/gpt-5.5
|
||||
aggregator:
|
||||
provider: openrouter
|
||||
model: anthropic/claude-opus-4.8
|
||||
reference_max_tokens: 600 # concise advice → faster turns
|
||||
```
|
||||
|
||||
Leave it unset (or `0`/blank) to keep the prior uncapped behavior.
|
||||
|
||||
## Terminal preset management
|
||||
|
||||
```bash
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue