mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-07-23 16:36:23 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 7

feat(browser): store full snapshots on truncation; make eval denylist opt-in (#65923)

Browse source 
* feat(browser): store full snapshots on truncation; make eval denylist opt-in

Two harness fixes motivated by BU_Bench results where fixed-verb + lossy
observation cost Hermes heavily vs code-driven browser agents:

1. Snapshot truncation no longer loses content. When a snapshot exceeds
   the 8000-char threshold, the complete accessibility tree is saved to
   cache/web (same truncate-and-store pattern as web_extract) and the
   truncated view / LLM summary includes the file path plus a ready-made
   read_file call. Element refs beyond the cut are recoverable without
   re-snapshotting. Stored copies are force-redacted and capped at 2MB;
   content-hash filenames dedupe repeated snapshots of the same page.

2. The browser_console(expression=...) sensitive-primitive denylist is
   now opt-in via browser.restrict_evaluate (default false). The
   names-based denylist blocked legitimate DOM extraction — any selector
   or expression containing 'fetch', 'cookie', 'input', etc. — which
   crippled the agent's only programmatic page-inspection path. The
   SSRF/private-URL egress guards in _browser_eval are independent of
   this policy and remain always-on. browser.allow_unsafe_evaluate keeps
   its meaning (bypass the denylist) for configs that already set it.

* test: update None-guard test for stored-snapshot pointer in _extract_relevant_content

test_normal_content_returned pinned the exact return value; the summary
now carries a pointer to the stored full snapshot. Assert the summary
passes through and the pointer is present instead.

* feat(browser): align snapshot threshold with web_extract's 15k char budget

SNAPSHOT_SUMMARIZE_THRESHOLD 8000 -> 15000, matching
web_tools.DEFAULT_EXTRACT_CHAR_LIMIT so the snapshot and web_extract
truncate-and-store paths give the model the same per-page budget.
_truncate_snapshot's default max_chars now follows the constant.
Invariant test added; docs (en+zh) and CLI tip updated.
This commit is contained in:
Teknium 2026-07-16 23:41:26 -07:00 • committed by GitHub
parent 779019ef7d
commit 0f102fa4dc
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 262 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

150
tools/browser_tool.py
View file

@ -229,8 +229,17 @@ DEFAULT_COMMAND_TIMEOUT = 30
MIN_OPEN_TIMEOUT = 60
MIN_FIRST_OPEN_TIMEOUT = 120
# Max tokens for snapshot content before summarization
SNAPSHOT_SUMMARIZE_THRESHOLD = 8000
# Max chars for snapshot content before truncation/summarization. Aligned
# with web_tools.DEFAULT_EXTRACT_CHAR_LIMIT (15000) — the snapshot and
# web_extract paths share the same truncate-and-store pattern, so the model
# gets the same per-page budget from both.
SNAPSHOT_SUMMARIZE_THRESHOLD = 15000
# Hard ceiling on the full-snapshot file written to cache/web when a snapshot
# is truncated or LLM-summarized. Mirrors web_tools.MAX_STORED_TEXT_CHARS —
# the model only ever sees the truncated view; the stored copy exists for
# read_file paging and must not write unbounded bytes to disk.
MAX_STORED_SNAPSHOT_CHARS = 2_000_000
# Commands that legitimately return empty stdout (e.g. close, record).
_EMPTY_OK_COMMANDS: frozenset = frozenset({"close", "record"})
@ -1833,7 +1842,7 @@ BROWSER_TOOL_SCHEMAS = [
},
{
"name": "browser_snapshot",
"description": "Get a text-based snapshot of the current page's accessibility tree. Returns interactive elements with ref IDs (like @e1, @e2) for browser_click and browser_type. full=false (default): compact view with interactive elements. full=true: complete page content. Snapshots over 8000 chars are truncated or LLM-summarized. Requires browser_navigate first. Note: browser_navigate already returns a compact snapshot — use this to refresh after interactions that change the page, or with full=true for complete content.",
"description": "Get a text-based snapshot of the current page's accessibility tree. Returns interactive elements with ref IDs (like @e1, @e2) for browser_click and browser_type. full=false (default): compact view with interactive elements. full=true: complete page content. Snapshots over 15000 chars are truncated or LLM-summarized; when that happens the complete snapshot is saved to a file and the output includes its path so you can page through the rest with read_file. Requires browser_navigate first. Note: browser_navigate already returns a compact snapshot — use this to refresh after interactions that change the page, or with full=true for complete content.",
"parameters": {
"type": "object",
"properties": {
@ -2579,14 +2588,62 @@ def _run_browser_command(
return result
def _store_full_snapshot(snapshot_text: str) -> Optional[str]:
"""Write a full page snapshot to cache/web and return its absolute path.
Called whenever a snapshot exceeds SNAPSHOT_SUMMARIZE_THRESHOLD and the
model is about to receive a truncated or LLM-summarized view. Mirrors
``web_tools._store_full_text``: the file lands in the same cache/web
directory (mounted read-only into remote backends via
credential_files._CACHE_DIRS) so the agent's read_file/terminal tools can
page through the complete accessibility tree — including element refs that
the truncated view dropped — on any backend.
The stored copy is secret-redacted (same force-redaction boundary as
``_redact_browser_output``) since page-rendered API keys or tokens must
not be written to disk unmasked. The filename is keyed on a content hash,
so repeated snapshots of the same page state dedupe to one file. Returns
None on failure (storage is best-effort; the truncated view is still
returned to the model).
"""
try:
import hashlib
from hermes_constants import get_hermes_dir
from agent.redact import redact_sensitive_text
content = redact_sensitive_text(snapshot_text, force=True)
if len(content) > MAX_STORED_SNAPSHOT_CHARS:
content = (
content[:MAX_STORED_SNAPSHOT_CHARS]
+ f"\n\n[... stored copy truncated at {MAX_STORED_SNAPSHOT_CHARS:,} chars "
f"of {len(content):,} ...]"
)
cache_dir = get_hermes_dir("cache/web", "web_cache")
cache_dir.mkdir(parents=True, exist_ok=True)
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()[:10]
path = cache_dir / f"browser-snapshot-{digest}.txt"
path.write_text(content, encoding="utf-8")
return str(path)
except Exception as exc: # noqa: BLE001
logger.debug("Failed to store full browser snapshot: %s", exc)
return None
def _extract_relevant_content(
snapshot_text: str,
user_task: Optional[str] = None
) -> str:
"""Use LLM to extract relevant content from a snapshot based on the user's task.
Falls back to simple truncation when no auxiliary text model is configured.
The full snapshot is stored to cache/web first (summarization is lossy —
the pointer lets the agent read anything the summary dropped). Falls back
to simple truncation when no auxiliary text model is configured.
"""
stored_path = _store_full_snapshot(snapshot_text)
stored_note = (
f'\n\n[Summarized from a {len(snapshot_text):,}-char snapshot. Full snapshot '
f'saved to: {stored_path} — read it with read_file if anything is missing.]'
) if stored_path else ""
if user_task:
extraction_prompt = (
f"You are a content extractor for a browser automation agent.\n\n"
@ -2628,41 +2685,59 @@ def _extract_relevant_content(
if model:
call_kwargs["model"] = model
response = call_llm(**call_kwargs)
extracted = (response.choices[0].message.content or "").strip() or _truncate_snapshot(snapshot_text)
extracted = (response.choices[0].message.content or "").strip()
if not extracted:
# _truncate_snapshot stores its own pointer (dedupes to the same
# cache file by content hash), so return it without stored_note.
return _truncate_snapshot(snapshot_text)
# Redact any secrets the auxiliary LLM may have echoed back.
return redact_sensitive_text(extracted)
return redact_sensitive_text(extracted) + stored_note
except Exception:
return _truncate_snapshot(snapshot_text)
def _truncate_snapshot(snapshot_text: str, max_chars: int = 8000) -> str:
def _truncate_snapshot(snapshot_text: str, max_chars: int = SNAPSHOT_SUMMARIZE_THRESHOLD) -> str:
"""Structure-aware truncation for snapshots.
Cuts at line boundaries so that accessibility tree elements are never
split mid-line, and appends a note telling the agent how much was
omitted.
split mid-line. The full snapshot is saved to cache/web (same pattern as
web_extract's truncate-and-store) and the appended note tells the agent
exactly where the complete text lives and how to page through it with
read_file — element refs beyond the cut are in the file, not lost.
Args:
snapshot_text: The snapshot text to truncate
max_chars: Maximum characters to keep
Returns:
Truncated text with indicator if truncated
Truncated text with a stored-full-text pointer if truncated
"""
if len(snapshot_text) <= max_chars:
return snapshot_text
stored_path = _store_full_snapshot(snapshot_text)
lines = snapshot_text.split('\n')
result: list[str] = []
chars = 0
# Reserve space for the truncation note (the stored-path variant is the
# longer of the two). Clamp so tiny max_chars values still keep content.
reserve = min(110 + len(stored_path or ""), max_chars // 2)
for line in lines:
if chars + len(line) + 1 > max_chars - 80: # reserve space for note
if chars + len(line) + 1 > max_chars - reserve:
break
result.append(line)
chars += len(line) + 1
remaining = len(lines) - len(result)
if remaining > 0:
result.append(f'\n[... {remaining} more lines truncated, use browser_snapshot for full content]')
if stored_path:
next_line = len(result) + 1
result.append(
f'\n[... {remaining} more lines truncated — full snapshot: '
f'read_file path="{stored_path}" offset={next_line} limit=200]'
)
else:
result.append(f'\n[... {remaining} more lines truncated, use browser_snapshot for full content]')
return '\n'.join(result)
@ -3447,11 +3522,8 @@ _SENSITIVE_BROWSER_EVAL_TOKENS: tuple[tuple[str, str], ...] = (
def _allow_unsafe_browser_evaluate() -> bool:
"""Return whether sensitive browser JS evaluation is explicitly allowed.
``browser_console(expression=...)`` is useful for read-only DOM inspection,
but a malicious page or prompt injection can try to steer the agent into
evaluating code that reads cookies/storage/form values or performs network
exfiltration. Keep harmless expressions (``document.title`` etc.) working,
while requiring a config opt-in for the dangerous primitives.
When true, ``browser_console(expression=...)`` runs without the
sensitive-primitive denylist even if ``browser.restrict_evaluate`` is set.
"""
try:
from hermes_cli.config import read_raw_config
@ -3463,6 +3535,30 @@ def _allow_unsafe_browser_evaluate() -> bool:
return False
def _restrict_browser_evaluate() -> bool:
"""Return whether the sensitive-primitive eval denylist is enabled.
Off by default. ``browser_console(expression=...)`` is the agent's only
programmatic page-inspection path, and the denylist blocks the *names* of
common primitives (``fetch``, ``cookie``, ``querySelector(...input...)``)
rather than any actual exfiltration — which also blocks a large class of
legitimate DOM extraction (any selector or page script text containing
those words). Egress itself is still gated by the SSRF/private-URL guards
in ``_browser_eval`` regardless of this setting. Users who want the
strict vocabulary denylist (e.g. when browsing hostile pages with a
logged-in profile) opt in with ``browser.restrict_evaluate: true``;
``browser.allow_unsafe_evaluate: true`` overrides it back off.
"""
try:
from hermes_cli.config import read_raw_config
cfg = read_raw_config()
return is_truthy_value(cfg_get(cfg, "browser", "restrict_evaluate"), default=False)
except Exception as e:
logger.debug("Could not read browser.restrict_evaluate from config: %s", e)
return False
def _decode_js_string_literal(literal: str) -> str:
"""Best-effort decode of a JavaScript string literal for policy checks.
@ -3519,7 +3615,16 @@ def _risky_browser_eval_reason(expression: str) -> Optional[str]:
def _enforce_browser_eval_policy(expression: str) -> Optional[str]:
"""Fail closed for sensitive browser JS evaluation unless config opts in."""
"""Block sensitive browser JS evaluation when the opt-in denylist is on.
The denylist is opt-in (``browser.restrict_evaluate: true``) because it
gates on primitive *names*, which cripples legitimate DOM extraction —
see ``_restrict_browser_evaluate``. Network egress to private/internal
addresses is enforced separately in ``_browser_eval`` and does not depend
on this policy.
"""
if not _restrict_browser_evaluate():
return None
if _allow_unsafe_browser_evaluate():
return None
reason = _risky_browser_eval_reason(expression)
@ -3527,10 +3632,11 @@ def _enforce_browser_eval_policy(expression: str) -> Optional[str]:
return None
return (
"Blocked: browser_console(expression=...) tried to use sensitive browser "
f"JavaScript primitive ({reason}). Use browser_snapshot/browser_get_images/"
"browser_console without expression for normal inspection, or set "
"browser.allow_unsafe_evaluate: true in config.yaml only for trusted pages "
"when this access is explicitly required."
f"JavaScript primitive ({reason}) while browser.restrict_evaluate is "
"enabled. Use browser_snapshot/browser_get_images/browser_console "
"without expression for normal inspection, or set "
"browser.restrict_evaluate: false in config.yaml to allow "
"programmatic evaluation."
)