mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-30 19:09:28 +00:00
3a1bbfac61
Replays the actual tool schemas captured from Epic's UE 5.8 ModelContextProtocol + AllToolsets plugins (830 tools / 52 toolsets) as live registry tools with mocked editor responses, then benchmarks eager vs bare-bridge vs bridge+listing at two scales (62-tool editor subset, full 830) on Claude Opus 4.8 (1M ctx; eager at 830 does not fit any 200K model — first call requests ~266K tokens). Headline (full 830, mean per task, rescored): eager 8/8 at 810,578 input tokens ($4.05); bare bridge 16/16 at 160,844 ($0.80); listing 16/16 at 257,264 ($1.29). Frontier model erases the accuracy gap in every mode; cost is the differentiator. At 62 tools eager wins on cost — consistent with the auto-threshold design. Also parameterizes livetest harness model + listing_max_tokens via env/args (TS_UE_MODEL, TS_UE_SCALE, TS_UE_MODES, TS_UE_LISTING_MAX).
50 lines
No EOL
650 B
JSON
50 lines
No EOL
650 B
JSON
{
|
|
"editor|eager": {
|
|
"succ": 12,
|
|
"n": 12,
|
|
"api": 2.2,
|
|
"itok": 116979,
|
|
"usd": 0.58,
|
|
"sec": 9.0
|
|
},
|
|
"editor|bridge": {
|
|
"succ": 12,
|
|
"n": 12,
|
|
"api": 4.7,
|
|
"itok": 146979,
|
|
"usd": 0.73,
|
|
"sec": 18.7
|
|
},
|
|
"editor|listing": {
|
|
"succ": 12,
|
|
"n": 12,
|
|
"api": 5,
|
|
"itok": 163663,
|
|
"usd": 0.82,
|
|
"sec": 21.5
|
|
},
|
|
"full|eager": {
|
|
"succ": 8,
|
|
"n": 8,
|
|
"api": 2.5,
|
|
"itok": 810578,
|
|
"usd": 4.05,
|
|
"sec": 18.9
|
|
},
|
|
"full|bridge": {
|
|
"succ": 16,
|
|
"n": 16,
|
|
"api": 5.2,
|
|
"itok": 160844,
|
|
"usd": 0.8,
|
|
"sec": 22.5
|
|
},
|
|
"full|listing": {
|
|
"succ": 16,
|
|
"n": 16,
|
|
"api": 4.6,
|
|
"itok": 257264,
|
|
"usd": 1.29,
|
|
"sec": 23.8
|
|
}
|
|
} |