fix(display): strip standalone tool-call XML tags from visible text

Port from openclaw/openclaw#67318. Some open models (notably Gemma
variants served via OpenRouter) emit tool calls as XML blocks inside
assistant content instead of via the structured tool_calls field:

  <function name="read_file"><parameter name="path">/tmp/x</parameter></function>
  <tool_call>{"name":"x"}</tool_call>
  <function_calls>[{...}]</function_calls>

Left unstripped, this raw XML leaked to gateway users (Discord, Telegram,
Matrix, Feishu, Signal, WhatsApp, etc.) and the CLI, since hermes-agent's
existing reasoning-tag stripper handled only <think>/<thinking>/<thought>
variants.

Extend _strip_think_blocks (run_agent.py) and _strip_reasoning_tags
(cli.py) to cover:
  * <tool_call>, <tool_calls>, <tool_result>
  * <function_call>, <function_calls>
  * <function name="..."> ... </function> (Gemma-style)

The <function> variant is boundary-gated (only strips when the tag sits
at start-of-line or after sentence punctuation AND carries a name="..."
attribute) so prose mentions like 'Use <function> declarations in JS'
are preserved. Dangling <function name="..."> with no close is
intentionally left visible — matches OpenClaw's asymmetry so a truncated
streaming tail still reaches the user.

Tests: 9 new cases in TestStripThinkBlocks (run_agent) + 9 in new file
tests/run_agent/test_strip_reasoning_tags_cli.py. Covers Qwen-style
<tool_call>, Gemma-style <function name="...">, multi-line payloads,
prose preservation, stray close tags, dangling open tags, and mixed
reasoning+tool_call content.

Note: this port covers the post-streaming final-text path, which is what
gateway adapters and CLI display consume. Extending the per-delta stream
filter in gateway/stream_consumer.py to hide these tags live as they
stream is a separate follow-up; for now users may see raw XML briefly
during a stream before the final cleaned text replaces it.

Refs: openclaw/openclaw#67318
This commit is contained in:
Teknium 2026-04-19 17:22:26 -07:00 committed by Teknium
parent 64b61cc24b
commit c345ec9a63
4 changed files with 232 additions and 0 deletions

View file

@ -2523,6 +2523,20 @@ class AIAgent:
4. Tag variants: ``<think>``, ``<thinking>``, ``<reasoning>``,
``<REASONING_SCRATCHPAD>``, ``<thought>`` (Gemma 4), all
case-insensitive.
Additionally strips standalone tool-call XML blocks that some open
models (notably Gemma variants on OpenRouter) emit inside assistant
content instead of via the structured ``tool_calls`` field:
* ``<tool_call></tool_call>``
* ``<tool_calls></tool_calls>``
* ``<tool_result></tool_result>``
* ``<function_call></function_call>``
* ``<function_calls></function_calls>``
* ``<function name=""></function>`` (Gemma style)
Ported from openclaw/openclaw#67318. The ``<function>`` variant is
boundary-gated (only strips when the tag sits at start-of-line or
after punctuation and carries a ``name="..."`` attribute) so prose
mentions like "Use <function> in JavaScript" are preserved.
"""
if not content:
return ""
@ -2534,6 +2548,30 @@ class AIAgent:
content = re.sub(r'<reasoning>.*?</reasoning>', '', content, flags=re.DOTALL | re.IGNORECASE)
content = re.sub(r'<REASONING_SCRATCHPAD>.*?</REASONING_SCRATCHPAD>', '', content, flags=re.DOTALL | re.IGNORECASE)
content = re.sub(r'<thought>.*?</thought>', '', content, flags=re.DOTALL | re.IGNORECASE)
# 1b. Tool-call XML blocks (openclaw/openclaw#67318). Handle the
# generic tag names first — they have no attribute gating since
# a literal <tool_call> in prose is already vanishingly rare.
for _tc_name in ("tool_call", "tool_calls", "tool_result",
"function_call", "function_calls"):
content = re.sub(
rf'<{_tc_name}\b[^>]*>.*?</{_tc_name}>',
'',
content,
flags=re.DOTALL | re.IGNORECASE,
)
# 1c. <function name="...">...</function> — Gemma-style standalone
# tool call. Only strip when the tag sits at a block boundary
# (start of text, after a newline, or after sentence-ending
# punctuation) AND carries a name="..." attribute. This keeps
# prose mentions like "Use <function> to declare" safe.
content = re.sub(
r'(?:(?<=^)|(?<=[\n\r.!?:]))[ \t]*'
r'<function\b[^>]*\bname\s*=[^>]*>'
r'(?:(?:(?!</function>).)*)</function>',
'',
content,
flags=re.DOTALL | re.IGNORECASE,
)
# 2. Unterminated reasoning block — open tag at a block boundary
# (start of text, or after a newline) with no matching close.
# Strip from the tag to end of string. Fixes #8878 / #9568
@ -2551,6 +2589,16 @@ class AIAgent:
content,
flags=re.IGNORECASE,
)
# 3b. Stray tool-call closers. (We do NOT strip bare <function> or
# unterminated <function name="..."> because a truncated tail
# during streaming may still be valuable to the user; matches
# OpenClaw's intentional asymmetry.)
content = re.sub(
r'</(?:tool_call|tool_calls|tool_result|function_call|function_calls|function)>\s*',
'',
content,
flags=re.IGNORECASE,
)
return content
@staticmethod