mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-23 16:36:23 +00:00
feat(tools): tiered tool disclosure — always defer MCP/plugin tools, scale the listing with catalog size
Tier 0: no MCP/plugin tools -> everything eager (pass-through).
Tier 1: deferred tools whose catalog listing fits min(threshold_pct%
of context, listing_max_tokens) -> bridge + skills-style
listing, degrading to names-only over budget.
Tier 2: listing over budget even names-only (Cloudflare's flat API
surface: 3,320 tools, names alone ~32K tokens) -> bare bridge,
discovery through tool_search only.
The old activation threshold (defer only when schemas > threshold_pct
of context) let mid-size catalogs ride eager and pay full schema cost;
with servers like Cloudflare (~597K tokens of schema, would not even
fit a 200K window) the binary gate is the wrong shape. Activation is
now driven purely by deferrable-tool presence; threshold_pct is
repurposed as the listing budget's context-relative leg.
- AssemblyResult gains tier + listing_form for observability
- listing_max_tokens default 4000 -> 20000 (cap 60000) so an 830-tool
catalog keeps a names-only listing while Cloudflare-scale drops to
bare bridge
- E2E verified against real captures: Linear 24 tools -> tier 1 full,
Epic UE 5.8 830 tools -> tier 1 names-only, Cloudflare 3,320 tools
-> tier 2 (both 200K and 1M context)
This commit is contained in:
Teknium
parent
d799184ede
commit
c80ce5aae2
5 changed files with 213 additions and 96 deletions
|
|
@ -2942,16 +2942,20 @@ DEFAULT_CONFIG = {
|
|||
# openclaw-tool-search-report PDF in this PR for the rationale.
|
||||
"tools": {
|
||||
"tool_search": {
|
||||
# "auto" (default) — activate only when deferrable tool schemas
|
||||
# exceed ``threshold_pct`` of the active model's context length,
|
||||
# so small toolsets pay no overhead.
|
||||
# "on" — always activate when there is at least one deferrable
|
||||
# tool. Use when you have many MCP servers and want maximum
|
||||
# token reduction unconditionally.
|
||||
# Tiered disclosure: any deferrable (MCP/plugin) tool activates
|
||||
# the bridge; the listing then scales with catalog size.
|
||||
# Tier 0 — no MCP/plugin tools: everything stays eager.
|
||||
# Tier 1 — catalog listing fits the budget: bridge + skills-style
|
||||
# name+description manifest (degrades to names-only).
|
||||
# Tier 2 — listing over budget even names-only (e.g. Cloudflare's
|
||||
# ~3,300-tool flat API surface): bare bridge, discovery through
|
||||
# tool_search only.
|
||||
# "auto"/"on" — activate when at least one deferrable tool exists.
|
||||
# "off" — disable entirely. Tools-array assembly is a pass-through.
|
||||
"enabled": "auto",
|
||||
# Percentage of context length at which "auto" mode kicks in.
|
||||
# 10 matches the Claude Code default. Range 0..100.
|
||||
# Listing budget as a percentage of the active model's context
|
||||
# length. Effective budget = min(this % of context,
|
||||
# listing_max_tokens). 10 matches the Claude Code default. Range 0..100.
|
||||
"threshold_pct": 10,
|
||||
# When the model calls tool_search without a ``limit`` argument,
|
||||
# how many hits to return. Range 1..max_search_limit.
|
||||
|
|
@ -2962,14 +2966,14 @@ DEFAULT_CONFIG = {
|
|||
# description: every deferred tool's name + first sentence of its
|
||||
# description (≤60 chars), grouped by MCP server / toolset. Keeps
|
||||
# capabilities discoverable while schemas stay deferred.
|
||||
# "auto" (default) — include when the listing fits listing_max_tokens
|
||||
# (falls back to names-only, then to the bare count).
|
||||
# "auto" (default) — include when the listing fits the budget
|
||||
# (falls back to names-only, then to the bare tier-2 bridge).
|
||||
# "on" — same rendering, but explicit intent to always list.
|
||||
# "off" — legacy bare-count bridge description.
|
||||
# "off" — always the bare bridge (tier 2 for every catalog).
|
||||
"listing": "auto",
|
||||
# Token budget for the embedded listing (chars/4 estimate).
|
||||
# Range 200..20000.
|
||||
"listing_max_tokens": 4000,
|
||||
# Absolute cap on the embedded listing in tokens (chars/4
|
||||
# estimate), regardless of context size. Range 200..60000.
|
||||
"listing_max_tokens": 20000,
|
||||
},
|
||||
},
|
||||
|
||||
|
|
|
|||
|
|
@ -555,10 +555,13 @@ def _compute_tool_definitions(
|
|||
config=ts_cfg,
|
||||
)
|
||||
if assembly.activated and not quiet_mode:
|
||||
_forms = {"full": "catalog listing embedded",
|
||||
"names": "names-only listing embedded",
|
||||
"none": "no listing (search-only)"}
|
||||
print(
|
||||
f"🔎 Tool Search: {assembly.deferred_count} MCP/plugin tools deferred "
|
||||
f"(~{assembly.deferred_tokens} tokens) behind tool_search/describe/call. "
|
||||
f"Threshold ~{assembly.threshold_tokens} tokens."
|
||||
f"🔎 Tool Search (tier {assembly.tier}): {assembly.deferred_count} "
|
||||
f"MCP/plugin tools deferred (~{assembly.deferred_tokens} tokens) behind "
|
||||
f"tool_search/describe/call — {_forms.get(assembly.listing_form, assembly.listing_form)}."
|
||||
)
|
||||
filtered_tools = assembly.tool_defs
|
||||
except Exception as e: # pragma: no cover — never break tool loading
|
||||
|
|
|
|||
|
|
@ -149,24 +149,27 @@ class TestThresholdGate:
|
|||
cfg = ToolSearchConfig.from_raw({"enabled": "on"})
|
||||
assert should_activate(cfg, deferrable_tokens=100, context_length=200_000)
|
||||
|
||||
def test_auto_below_threshold_does_not_activate(self):
|
||||
def test_auto_activates_with_any_deferrable(self):
|
||||
"""Tiered disclosure: ANY deferrable tool activates the bridge —
|
||||
the threshold now bounds the listing, not activation."""
|
||||
from tools.tool_search import ToolSearchConfig, should_activate
|
||||
cfg = ToolSearchConfig.from_raw({"enabled": "auto", "threshold_pct": 10})
|
||||
# 5% of 200K = below 10% threshold
|
||||
assert not should_activate(cfg, deferrable_tokens=10_000, context_length=200_000)
|
||||
|
||||
def test_auto_at_or_above_threshold_activates(self):
|
||||
from tools.tool_search import ToolSearchConfig, should_activate
|
||||
cfg = ToolSearchConfig.from_raw({"enabled": "auto", "threshold_pct": 10})
|
||||
assert should_activate(cfg, deferrable_tokens=20_000, context_length=200_000)
|
||||
assert should_activate(cfg, deferrable_tokens=100, context_length=200_000)
|
||||
assert should_activate(cfg, deferrable_tokens=50_000, context_length=200_000)
|
||||
# unknown context length: still activates
|
||||
assert should_activate(cfg, deferrable_tokens=100, context_length=0)
|
||||
|
||||
def test_auto_without_context_length_uses_20k_cutoff(self):
|
||||
"""Fallback cutoff used when the active model is unknown."""
|
||||
from tools.tool_search import ToolSearchConfig, should_activate
|
||||
cfg = ToolSearchConfig.from_raw({"enabled": "auto"})
|
||||
assert not should_activate(cfg, deferrable_tokens=10_000, context_length=0)
|
||||
assert should_activate(cfg, deferrable_tokens=25_000, context_length=0)
|
||||
def test_listing_budget_min_of_pct_and_cap(self):
|
||||
from tools.tool_search import ToolSearchConfig, listing_token_budget
|
||||
cfg = ToolSearchConfig.from_raw(
|
||||
{"threshold_pct": 10, "listing_max_tokens": 8000})
|
||||
# 10% of 200K = 20K > cap 8K → cap wins
|
||||
assert listing_token_budget(cfg, 200_000) == 8000
|
||||
# 10% of 50K = 5K < cap 8K → pct leg wins
|
||||
assert listing_token_budget(cfg, 50_000) == 5000
|
||||
# unknown context → 20K fallback for the pct leg, still capped
|
||||
assert listing_token_budget(cfg, 0) == 8000
|
||||
assert listing_token_budget(cfg, None) == 8000
|
||||
|
||||
def test_token_estimate_proportional_to_schema_size(self):
|
||||
from tools.tool_search import estimate_tokens_from_schemas
|
||||
|
|
@ -250,20 +253,65 @@ class TestAssembly:
|
|||
assert not result.activated
|
||||
assert {t["function"]["name"] for t in result.tool_defs} == {"terminal", "read_file"}
|
||||
|
||||
def test_below_threshold_returns_unchanged(self):
|
||||
"""Tiny deferrable surface: don't bother."""
|
||||
@staticmethod
|
||||
def _register_mcp(name):
|
||||
from tools.registry import registry
|
||||
|
||||
def _handler(args, task_id=None, **kw):
|
||||
return json.dumps({"ok": True})
|
||||
|
||||
registry.register(
|
||||
name=name,
|
||||
handler=_handler,
|
||||
schema=_td(name, "Deferred capability description.")["function"],
|
||||
toolset="mcp-tiertest",
|
||||
)
|
||||
|
||||
def test_small_deferrable_surface_defers_with_full_listing(self):
|
||||
"""Tiered disclosure: even a tiny MCP/plugin surface defers (tier 1),
|
||||
with the full name+description listing embedded."""
|
||||
from tools.tool_search import assemble_tool_defs, ToolSearchConfig
|
||||
# _td renders to ~80 chars / 20 tokens. 3 of them = ~60 tokens.
|
||||
# 10% of 200K = 20K. Way below.
|
||||
defs = [_td("unknown_tool_a"), _td("unknown_tool_b"), _td("unknown_tool_c")]
|
||||
for n in ("tier_small_a", "tier_small_b", "tier_small_c"):
|
||||
self._register_mcp(n)
|
||||
defs = [_td("terminal", "Run shell")] + [
|
||||
_td(n, "Deferred capability description.")
|
||||
for n in ("tier_small_a", "tier_small_b", "tier_small_c")]
|
||||
result = assemble_tool_defs(
|
||||
defs,
|
||||
context_length=200_000,
|
||||
config=ToolSearchConfig.from_raw({"enabled": "auto", "threshold_pct": 10}),
|
||||
)
|
||||
assert not result.activated
|
||||
assert result.activated
|
||||
assert result.tier == 1
|
||||
assert result.listing_form == "full"
|
||||
names = {(t.get("function") or {}).get("name") for t in result.tool_defs}
|
||||
assert "tool_search" not in names
|
||||
assert "tool_search" in names
|
||||
assert "terminal" in names # core stays eager
|
||||
search = next(t for t in result.tool_defs
|
||||
if t["function"]["name"] == "tool_search")
|
||||
assert "tier_small_a" in search["function"]["description"]
|
||||
|
||||
def test_oversized_catalog_degrades_to_bare_bridge_tier2(self):
|
||||
"""When even the names-only listing exceeds the budget, tier 2:
|
||||
bare bridge, no listing."""
|
||||
from tools.tool_search import assemble_tool_defs, ToolSearchConfig
|
||||
names = [f"tier2_very_long_tool_name_number_{i:04d}_extra" for i in range(400)]
|
||||
for n in names:
|
||||
self._register_mcp(n)
|
||||
defs = [_td(n, "A description that will not matter at this size.")
|
||||
for n in names]
|
||||
result = assemble_tool_defs(
|
||||
defs,
|
||||
context_length=200_000,
|
||||
config=ToolSearchConfig.from_raw(
|
||||
{"enabled": "auto", "threshold_pct": 10, "listing_max_tokens": 200}),
|
||||
)
|
||||
assert result.activated
|
||||
assert result.tier == 2
|
||||
assert result.listing_form == "none"
|
||||
search = next(t for t in result.tool_defs
|
||||
if t["function"]["name"] == "tool_search")
|
||||
assert "tier2_very_long_tool_name_number_0000" not in search["function"]["description"]
|
||||
|
||||
def test_idempotent_when_bridge_already_present(self):
|
||||
from tools.tool_search import assemble_tool_defs, ToolSearchConfig, BRIDGE_TOOL_NAMES
|
||||
|
|
@ -548,7 +596,7 @@ class TestCatalogListing:
|
|||
from tools.tool_search import ToolSearchConfig
|
||||
cfg = ToolSearchConfig.from_raw(None)
|
||||
assert cfg.listing == "auto"
|
||||
assert cfg.listing_max_tokens == 4000
|
||||
assert cfg.listing_max_tokens == 20000
|
||||
# legacy bool shapes keep defaults too
|
||||
assert ToolSearchConfig.from_raw(True).listing == "auto"
|
||||
|
||||
|
|
@ -556,7 +604,7 @@ class TestCatalogListing:
|
|||
from tools.tool_search import ToolSearchConfig
|
||||
cfg = ToolSearchConfig.from_raw({"listing": "off", "listing_max_tokens": 999999})
|
||||
assert cfg.listing == "off"
|
||||
assert cfg.listing_max_tokens == 20000
|
||||
assert cfg.listing_max_tokens == 60000
|
||||
cfg2 = ToolSearchConfig.from_raw({"listing": "garbage", "listing_max_tokens": -5})
|
||||
assert cfg2.listing == "auto"
|
||||
assert cfg2.listing_max_tokens == 200
|
||||
|
|
|
|||
|
|
@ -9,9 +9,17 @@ for the full rationale):
|
|||
|
||||
* Core tools defined in ``toolsets._HERMES_CORE_TOOLS`` are *never* deferred.
|
||||
Always-load means always-load. No exceptions.
|
||||
* The threshold gate runs every assembly: when deferrable tools would consume
|
||||
less than ``threshold_pct`` of the model's context window (default 10%),
|
||||
tool search is a no-op and the tools array passes through unchanged.
|
||||
* Tiered disclosure (July 2026 plan): the moment ANY deferrable (MCP/plugin)
|
||||
tools are present, they hide behind the bridge. What scales with catalog
|
||||
size is the *listing*, not the activation decision:
|
||||
- Tier 0 — no MCP/plugin tools: pure passthrough, everything eager.
|
||||
- Tier 1 — deferred tools whose catalog listing fits the listing budget
|
||||
(``min(threshold_pct`` of context, ``listing_max_tokens)``): bridge +
|
||||
skills-style listing (name + short description per tool), degrading to
|
||||
a names-only listing when the full form is over budget.
|
||||
- Tier 2 — listing over budget even names-only (e.g. Cloudflare's flat
|
||||
API surface, ~3,300 tools whose names alone are ~32K tokens): bare
|
||||
bridge; tools are discoverable only through ``tool_search``.
|
||||
* The catalog is stateless across turns and tools-array assemblies. It is
|
||||
rebuilt from the current tool-defs list every time. This is the lesson
|
||||
from OpenClaw's cron regression (openclaw/openclaw#84141): a session-keyed
|
||||
|
|
@ -65,17 +73,25 @@ class ToolSearchConfig:
|
|||
"""Resolved, validated tool-search configuration for a single assembly."""
|
||||
|
||||
enabled: str # "auto" | "on" | "off"
|
||||
threshold_pct: float # 0..100 — only used when enabled == "auto"
|
||||
# Listing budget as a percentage of the model's context window. Under
|
||||
# tiered disclosure this no longer gates *activation* (any deferrable
|
||||
# tool activates the bridge) — it bounds how much context the embedded
|
||||
# catalog listing may consume before disclosure degrades:
|
||||
# full listing -> names-only -> bare bridge (tier 2).
|
||||
threshold_pct: float # 0..100
|
||||
search_default_limit: int
|
||||
max_search_limit: int
|
||||
# Catalog listing ("skills-style" progressive disclosure): when active,
|
||||
# a grouped name + short-description manifest of every deferred tool is
|
||||
# embedded in the tool_search bridge description, so capabilities stay
|
||||
# DISCOVERABLE (like the skills listing in the system prompt) while full
|
||||
# schemas stay deferred. "auto" = include when it fits listing_max_tokens;
|
||||
# "on" = always include; "off" = legacy bare-count behavior.
|
||||
# schemas stay deferred. "auto" = include when it fits the listing
|
||||
# budget (falls back to names-only, then to none = bare bridge);
|
||||
# "on" = same rendering, explicit intent; "off" = always bare bridge.
|
||||
listing: str = "auto" # "auto" | "on" | "off"
|
||||
listing_max_tokens: int = 4000
|
||||
# Absolute cap on the embedded listing, regardless of context size.
|
||||
# Effective budget = min(listing_max_tokens, threshold_pct% of context).
|
||||
listing_max_tokens: int = 20000
|
||||
|
||||
@classmethod
|
||||
def from_raw(cls, raw: Any) -> "ToolSearchConfig":
|
||||
|
|
@ -123,7 +139,7 @@ class ToolSearchConfig:
|
|||
listing = listing_raw
|
||||
else:
|
||||
listing = "auto"
|
||||
listing_max_tokens = max(200, min(20000, _safe_int(raw.get("listing_max_tokens"), 4000)))
|
||||
listing_max_tokens = max(200, min(60000, _safe_int(raw.get("listing_max_tokens"), 20000)))
|
||||
|
||||
return cls(
|
||||
enabled=enabled,
|
||||
|
|
@ -259,24 +275,37 @@ def should_activate(
|
|||
) -> bool:
|
||||
"""Decide whether tool search should activate for the current assembly.
|
||||
|
||||
``"off"`` skips unconditionally. ``"on"`` activates unconditionally
|
||||
(as long as there is at least one deferrable tool — there's no point
|
||||
swapping a no-op). ``"auto"`` activates when the deferrable schemas
|
||||
would consume ``threshold_pct`` of context or more.
|
||||
``"off"`` skips unconditionally. ``"on"`` and ``"auto"`` activate whenever
|
||||
at least one deferrable tool exists (there's no point swapping a no-op).
|
||||
|
||||
Tiered-disclosure semantics (July 2026): the presence of ANY MCP/plugin
|
||||
tool activates the bridge — schemas always defer. What the threshold now
|
||||
controls is the *listing budget* (see :func:`listing_token_budget`), not
|
||||
activation. ``context_length`` is retained in the signature for
|
||||
backward compatibility with existing callers.
|
||||
"""
|
||||
if config.enabled == "off":
|
||||
return False
|
||||
if deferrable_tokens <= 0:
|
||||
return False
|
||||
if config.enabled == "on":
|
||||
return True
|
||||
# auto
|
||||
if not context_length or context_length <= 0:
|
||||
# Without a known context size, fall back to a fixed 20K-token cutoff
|
||||
# — the cliff above which Anthropic and OpenAI both saw quality drops.
|
||||
return deferrable_tokens >= 20_000
|
||||
threshold_tokens = int(context_length * (config.threshold_pct / 100.0))
|
||||
return deferrable_tokens >= threshold_tokens
|
||||
return True
|
||||
|
||||
|
||||
def listing_token_budget(
|
||||
config: ToolSearchConfig,
|
||||
context_length: Optional[int],
|
||||
) -> int:
|
||||
"""Effective token budget for the embedded catalog listing.
|
||||
|
||||
``min(listing_max_tokens, threshold_pct% of context)``. Without a known
|
||||
context size, the percentage leg falls back to the fixed 20K cutoff the
|
||||
activation gate historically used (10% of a typical 200K window).
|
||||
"""
|
||||
if context_length and context_length > 0:
|
||||
pct_leg = int(context_length * (config.threshold_pct / 100.0))
|
||||
else:
|
||||
pct_leg = 20_000
|
||||
return max(0, min(config.listing_max_tokens, pct_leg))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -479,7 +508,7 @@ def _listing_group_label(source_name: str) -> str:
|
|||
def build_catalog_listing(
|
||||
deferrable: List[Dict[str, Any]],
|
||||
*,
|
||||
max_tokens: int = 4000,
|
||||
max_tokens: int = 20000,
|
||||
) -> Optional[str]:
|
||||
"""Render a skills-style manifest of the deferred catalog.
|
||||
|
||||
|
|
@ -502,8 +531,23 @@ def build_catalog_listing(
|
|||
2. names-only listing, still grouped
|
||||
3. ``None`` — caller falls back to the legacy bare-count description
|
||||
"""
|
||||
text, _form = build_catalog_listing_with_form(deferrable, max_tokens=max_tokens)
|
||||
return text
|
||||
|
||||
|
||||
def build_catalog_listing_with_form(
|
||||
deferrable: List[Dict[str, Any]],
|
||||
*,
|
||||
max_tokens: int = 20000,
|
||||
) -> Tuple[Optional[str], str]:
|
||||
"""Like :func:`build_catalog_listing` but also reports the form used.
|
||||
|
||||
Returns ``(text, form)`` where ``form`` is ``"full"`` (names + short
|
||||
descriptions), ``"names"`` (names-only fallback), or ``"none"`` (over
|
||||
budget in every form — tier-2 bare bridge).
|
||||
"""
|
||||
if not deferrable:
|
||||
return None
|
||||
return None, "none"
|
||||
|
||||
groups: Dict[str, List[Tuple[str, str]]] = {}
|
||||
for td in deferrable:
|
||||
|
|
@ -516,7 +560,7 @@ def build_catalog_listing(
|
|||
groups.setdefault(label, []).append((name, _short_desc(fn.get("description", ""))))
|
||||
|
||||
if not groups:
|
||||
return None
|
||||
return None, "none"
|
||||
|
||||
def render(with_descriptions: bool) -> str:
|
||||
lines: List[str] = ["Deferred tool catalog (call schemas via "
|
||||
|
|
@ -531,11 +575,11 @@ def build_catalog_listing(
|
|||
lines.append(", ".join(name for name, _ in tools))
|
||||
return "\n".join(lines)
|
||||
|
||||
for with_desc in (True, False):
|
||||
for with_desc, form in ((True, "full"), (False, "names")):
|
||||
text = render(with_desc)
|
||||
if math.ceil(len(text) / CHARS_PER_TOKEN) <= max_tokens:
|
||||
return text
|
||||
return None
|
||||
return text, form
|
||||
return None, "none"
|
||||
|
||||
|
||||
def bridge_tool_schemas(
|
||||
|
|
@ -655,6 +699,12 @@ class AssemblyResult:
|
|||
deferred_count: int = 0
|
||||
deferred_tokens: int = 0
|
||||
threshold_tokens: int = 0
|
||||
# Disclosure tier actually applied:
|
||||
# 0 = passthrough (no deferrable tools, or tool_search off)
|
||||
# 1 = bridge + catalog listing (full or names-only)
|
||||
# 2 = bare bridge — catalog too large for any listing form
|
||||
tier: int = 0
|
||||
listing_form: str = "none" # "full" | "names" | "none"
|
||||
|
||||
|
||||
def assemble_tool_defs(
|
||||
|
|
@ -694,19 +744,24 @@ def assemble_tool_defs(
|
|||
deferred_count=len(deferrable),
|
||||
deferred_tokens=deferrable_tokens,
|
||||
threshold_tokens=int((context_length or 0) * (config.threshold_pct / 100.0)),
|
||||
tier=0,
|
||||
)
|
||||
|
||||
listing = None
|
||||
listing_form = "none"
|
||||
listing_budget = listing_token_budget(config, context_length)
|
||||
if config.listing != "off":
|
||||
listing = build_catalog_listing(deferrable, max_tokens=config.listing_max_tokens)
|
||||
listing, listing_form = build_catalog_listing_with_form(
|
||||
deferrable, max_tokens=listing_budget)
|
||||
bridge = bridge_tool_schemas(len(deferrable), listing=listing)
|
||||
result = visible + bridge
|
||||
threshold_tokens = int((context_length or 0) * (config.threshold_pct / 100.0))
|
||||
tier = 1 if listing else 2
|
||||
|
||||
logger.info(
|
||||
"tool_search activated: %d core/visible tools kept, %d deferred (~%d tokens, threshold ~%d), catalog listing %s",
|
||||
len(visible), len(deferrable), deferrable_tokens, threshold_tokens,
|
||||
"embedded" if listing else "omitted",
|
||||
"tool_search activated (tier %d): %d core/visible tools kept, %d deferred "
|
||||
"(~%d tokens), listing %s (budget ~%d tokens)",
|
||||
tier, len(visible), len(deferrable), deferrable_tokens,
|
||||
listing_form, listing_budget,
|
||||
)
|
||||
|
||||
return AssemblyResult(
|
||||
|
|
@ -714,7 +769,9 @@ def assemble_tool_defs(
|
|||
activated=True,
|
||||
deferred_count=len(deferrable),
|
||||
deferred_tokens=deferrable_tokens,
|
||||
threshold_tokens=threshold_tokens,
|
||||
threshold_tokens=listing_budget,
|
||||
tier=tier,
|
||||
listing_form=listing_form,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -860,6 +917,8 @@ __all__ = [
|
|||
"should_activate",
|
||||
"build_catalog",
|
||||
"build_catalog_listing",
|
||||
"build_catalog_listing_with_form",
|
||||
"listing_token_budget",
|
||||
"search_catalog",
|
||||
"bridge_tool_schemas",
|
||||
"assemble_tool_defs",
|
||||
|
|
|
|||
|
|
@ -55,19 +55,20 @@ see the underlying tool, not the bridge.
|
|||
|
||||
## When does it activate?
|
||||
|
||||
By default Tool Search runs in `auto` mode: it activates only when the
|
||||
deferrable tool schemas would consume at least 10% of the active model's
|
||||
context window. Below that, the tools-array assembly is a pure
|
||||
pass-through and you pay no overhead.
|
||||
Tool Search uses **tiered disclosure**: the presence of *any* deferrable
|
||||
(MCP/plugin) tool activates the bridge; what scales with catalog size is
|
||||
how much of the catalog stays visible, not whether schemas defer.
|
||||
|
||||
This decision is re-evaluated every time the tools array is built, so:
|
||||
| Tier | Condition | What the model sees |
|
||||
| --- | --- | --- |
|
||||
| **0** | No MCP/plugin tools | Every tool eager, no bridge. Pass-through. |
|
||||
| **1** | Deferred catalog's listing fits the budget | Bridge + a skills-style manifest of every deferred tool (name + short description, degrading to names-only when over budget). |
|
||||
| **2** | Listing exceeds the budget even names-only (e.g. Cloudflare's flat API surface: ~3,300 tools whose names alone are ~32K tokens) | Bare bridge — tools are discoverable only through `tool_search`. |
|
||||
|
||||
- A session with just a few MCP tools and a long context model never
|
||||
activates Tool Search.
|
||||
- A session with many MCP servers attached (15+ tools typically) starts
|
||||
activating it.
|
||||
- Removing MCP servers mid-session correctly returns to direct exposure
|
||||
on the next assembly.
|
||||
The listing budget is `min(threshold_pct% of context, listing_max_tokens)`.
|
||||
The decision is re-evaluated every time the tools array is built, so
|
||||
adding or removing MCP servers mid-session moves the session between
|
||||
tiers on the next assembly.
|
||||
|
||||
## Configuration
|
||||
|
||||
|
|
@ -75,21 +76,21 @@ This decision is re-evaluated every time the tools array is built, so:
|
|||
tools:
|
||||
tool_search:
|
||||
enabled: auto # auto (default), on, or off
|
||||
threshold_pct: 10 # percentage of context — only used in auto mode
|
||||
threshold_pct: 10 # listing budget as a percentage of context
|
||||
search_default_limit: 5
|
||||
max_search_limit: 20
|
||||
listing: auto # embed a grouped name+description catalog manifest
|
||||
listing_max_tokens: 4000
|
||||
listing_max_tokens: 20000
|
||||
```
|
||||
|
||||
| Key | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `enabled` | `auto` | `auto` activates above threshold; `on` always activates if there's at least one deferrable tool; `off` disables entirely. |
|
||||
| `threshold_pct` | `10` | Percentage of context length at which `auto` mode kicks in. Range 0–100. |
|
||||
| `enabled` | `auto` | `auto`/`on` activate whenever at least one deferrable tool exists; `off` disables entirely (everything stays eager). |
|
||||
| `threshold_pct` | `10` | Listing budget as a percentage of the active model's context length. Range 0–100. |
|
||||
| `search_default_limit` | `5` | Hits returned when the model calls `tool_search` without a `limit`. |
|
||||
| `max_search_limit` | `20` | Hard upper bound the model can request via `limit`. Range 1–50. |
|
||||
| `listing` | `auto` | Embed a skills-style manifest of every deferred tool (name + first sentence of its description, ≤60 chars, grouped by MCP server) in the `tool_search` bridge description. `auto` includes it when it fits the budget (falling back to names-only, then to a bare count); `on`/`off` force either way. |
|
||||
| `listing_max_tokens` | `4000` | Token budget for the embedded listing. Range 200–20000. |
|
||||
| `listing` | `auto` | Embed a skills-style manifest of every deferred tool (name + first sentence of its description, ≤60 chars, grouped by MCP server) in the `tool_search` bridge description. `auto` includes it when it fits the budget (falling back to names-only, then to the bare tier-2 bridge); `on`/`off` force either way. |
|
||||
| `listing_max_tokens` | `20000` | Absolute cap on the embedded listing, regardless of context size. Range 200–60000. |
|
||||
|
||||
### Why the listing exists
|
||||
|
||||
|
|
@ -112,13 +113,15 @@ tools:
|
|||
## When NOT to use it
|
||||
|
||||
Tool Search trades a fixed per-turn token cost (the three bridge tool
|
||||
schemas, ~300 tokens) and at least one extra round trip (search →
|
||||
describe → call) for the savings on the deferred schemas. It's a clear
|
||||
win when you have many tools and use few per turn; it's overhead when
|
||||
you have few tools total.
|
||||
schemas plus the catalog listing) and at least one extra round trip on
|
||||
cold tools (describe → call) for the savings on the deferred schemas.
|
||||
At tier 1 the listing keeps every capability visible, so the discovery
|
||||
round trip usually disappears — the model goes straight to
|
||||
`tool_describe`. Live benchmarking showed the listing mode matching
|
||||
eager loading's task success while costing less than the bare bridge.
|
||||
|
||||
The `auto` default handles this for you. If you set `enabled: on`
|
||||
unconditionally, expect a slight per-turn cost on small toolsets.
|
||||
If you want the old always-eager behavior for a small toolset, set
|
||||
`enabled: off`.
|
||||
|
||||
## Trade-offs that don't go away
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue