teknium1 3a1bbfac61 bench: Unreal-scale live benchmark — Epic's real 830 UE 5.8 schemas replayed (Opus 4.8) ... Replays the actual tool schemas captured from Epic's UE 5.8 ModelContextProtocol + AllToolsets plugins (830 tools / 52 toolsets) as live registry tools with mocked editor responses, then benchmarks eager vs bare-bridge vs bridge+listing at two scales (62-tool editor subset, full 830) on Claude Opus 4.8 (1M ctx; eager at 830 does not fit any 200K model — first call requests ~266K tokens). Headline (full 830, mean per task, rescored): eager 8/8 at 810,578 input tokens ($4.05); bare bridge 16/16 at 160,844 ($0.80); listing 16/16 at 257,264 ($1.29). Frontier model erases the accuracy gap in every mode; cost is the differentiator. At 62 tools eager wins on cost — consistent with the auto-threshold design. Also parameterizes livetest harness model + listing_max_tokens via env/args (TS_UE_MODEL, TS_UE_SCALE, TS_UE_MODES, TS_UE_LISTING_MAX).		2026-07-26 08:26:09 -07:00
..
ue_bench_rows.json	bench: Unreal-scale live benchmark — Epic's real 830 UE 5.8 schemas replayed (Opus 4.8)	2026-07-26 08:26:09 -07:00
ue_bench_summary.json	bench: Unreal-scale live benchmark — Epic's real 830 UE 5.8 schemas replayed (Opus 4.8)	2026-07-26 08:26:09 -07:00