mirrors/hermes-agent
1
0
Fork 0
mirror of https://github.com/NousResearch/hermes-agent.git synced 2026-05-22 05:22:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 10

feat(web): firecrawl plugin natively supports crawl; delete legacy inline path

Browse source 
The web-provider migration originally left firecrawl crawl as the only
provider-specific code remaining inline in tools/web_tools.py (~250
lines of Firecrawl-specific crawl orchestration that didn't fit the
plugin's existing surface). This commit closes that gap.

What this adds
--------------
1. plugins/web/firecrawl/provider.py: implement async ``crawl(url, **kwargs)``
   - Accepts the same kwargs as the dispatcher passes to any crawl
     provider (``instructions``, ``depth``, ``limit``); Firecrawl's
     /crawl endpoint ignores ``instructions`` and ``depth`` so we log
     and drop with a clear info message.
   - Wraps the sync SDK ``crawl()`` call in asyncio.to_thread so the
     gateway event loop isn't blocked on a multi-page crawl.
   - Preserves the response-shape normalization across pydantic /
     typed-object / dict variants that the legacy inline code did.
   - Preserves per-page website-policy re-check (catches blocked
     redirects after the SDK returns).
   - Returns the same {"results": [...]} shape so the dispatcher's
     shared LLM-summarization post-processing path works unchanged.
   - Sets supports_crawl() to True so the dispatcher routes through
     the plugin instead of the legacy fallthrough.

2. tools/web_tools.py: delete the entire legacy firecrawl crawl block
   that used to run after "No registered provider supports crawl" —
   ~270 lines including:
   - check_firecrawl_api_key gate + typed error
   - inline SSRF + website-policy seed-URL gate (dispatcher already
     does this)
   - Firecrawl client setup with crawl_params
   - 100+ lines of pydantic/dict/typed-object normalization
   - Per-page LLM-processing loop (kept in the dispatcher's shared
     post-processing path; that's where it always belonged)
   - trimming + base64 image cleanup (still done in the dispatcher's
     shared path)

   Replaced with a single typed-error branch when no crawl-capable
   provider is available: "web_crawl has no available backend. Set
   FIRECRAWL_API_KEY (or FIRECRAWL_API_URL for self-hosted), or set
   TAVILY_API_KEY for Tavily."

Test updates
------------
- tests/tools/test_website_policy.py:
  - test_web_crawl_short_circuits_blocked_url: dispatcher seed-URL
    gate still runs on web_tools.check_website_access (no change to
    that patch), but the firecrawl client lockdown moved to the
    plugin module — patch firecrawl_provider._get_firecrawl_client
    instead of web_tools._get_firecrawl_client. The dispatcher
    short-circuits before the plugin runs, so the test still passes.
  - test_web_crawl_blocks_redirected_final_url: patch the per-page
    policy gate at plugins.web.firecrawl.provider.check_website_access
    (where it now runs) AND on web_tools (where the seed-URL gate
    still runs). Patch firecrawl_provider._get_firecrawl_client for
    the FakeCrawlClient injection. Both checks flow through the same
    fake_check function.
- tests/plugins/web/test_web_search_provider_plugins.py:
  - Update parametrized capability-flag spec: firecrawl supports_crawl
    is now True.
  - Add test_firecrawl_crawl_returns_error_dict_when_unconfigured —
    verifies inspect.iscoroutinefunction(p.crawl) is True and that
    the async crawl returns a per-page error dict (not a raise) when
    FIRECRAWL_API_KEY is missing.

Verified
--------
- 218/218 web tests pass (was 173, +44 plugin tests + 1 new firecrawl
  crawl test from this commit = 218 with the test deduplication).
- Compile-clean (py_compile passes on both files).
- Provider capabilities matrix confirmed end-to-end:
    name        search  extract  crawl   async-extract?  async-crawl?
    firecrawl   True    True     True    True            True
    tavily      True    True     True    False           False
  Both crawl-capable providers exercise the dispatcher's
  inspect.iscoroutinefunction async-or-sync detection.

Net diff
--------
- tools/web_tools.py: -254 lines (legacy inline crawl gone)
- plugins/web/firecrawl/provider.py: +185 lines (crawl method)
- test_website_policy.py: +14/-9 lines (patch locations)
- test_web_search_provider_plugins.py: +22/-1 lines (capability flag
  + new firecrawl crawl test)
- Total: -32 net LoC; tools/web_tools.py is now 1509 lines (was 1763
  before this commit, 2227 before the migration started).
This commit is contained in:
kshitijk4poor 2026-05-14 01:37:57 +05:30 • committed by Teknium
parent e8cee87e85
commit 21e3a863bb
4 changed files with 243 additions and 275 deletions
Download patch file Download diff file Expand all files Collapse all files

284
tools/web_tools.py
View file

@ -1285,275 +1285,23 @@ async def web_crawl_tool(
_debug.save()
return cleaned_result
# No registered provider supports crawl. Fall through to the
# Firecrawl-via-summarize path below (legacy behavior) when
# Firecrawl credentials are configured.
# web_crawl requires Firecrawl or the Firecrawl tool-gateway — Parallel has no crawl API
if not check_firecrawl_api_key():
return json.dumps({
"error": "web_crawl requires Firecrawl. Set FIRECRAWL_API_KEY, FIRECRAWL_API_URL"
f"{_firecrawl_backend_help_suffix()}, or use web_search + web_extract instead.",
"success": False,
}, ensure_ascii=False)
# Ensure URL has protocol
if not url.startswith(('http://', 'https://')):
url = f'https://{url}'
logger.info("Added https:// prefix to URL: %s", url)
instructions_text = f" with instructions: '{instructions}'" if instructions else ""
logger.info("Crawling %s%s", url, instructions_text)
# SSRF protection — block private/internal addresses
if not is_safe_url(url):
return json.dumps({"results": [{"url": url, "title": "", "content": "",
"error": "Blocked: URL targets a private or internal network address"}]}, ensure_ascii=False)
# Website policy check — block before crawling
blocked = check_website_access(url)
if blocked:
logger.info("Blocked web_crawl for %s by rule %s", blocked["host"], blocked["rule"])
return json.dumps({"results": [{"url": url, "title": "", "content": "", "error": blocked["message"],
"blocked_by_policy": {"host": blocked["host"], "rule": blocked["rule"], "source": blocked["source"]}}]}, ensure_ascii=False)
# Use Firecrawl's v2 crawl functionality
# Docs: https://docs.firecrawl.dev/features/crawl
# The crawl() method automatically waits for completion and returns all data
# Build crawl parameters - keep it simple
crawl_params = {
"limit": 20, # Limit number of pages to crawl
"scrape_options": {
"formats": ["markdown"] # Just markdown for simplicity
}
}
# Note: The 'prompt' parameter is not documented for crawl
# Instructions are typically used with the Extract endpoint, not Crawl
if instructions:
logger.info("Instructions parameter ignored (not supported in crawl API)")
from tools.interrupt import is_interrupted as _is_int
if _is_int():
return tool_error("Interrupted", success=False)
try:
crawl_result = _get_firecrawl_client().crawl(
url=url,
**crawl_params
)
except Exception as e:
logger.debug("Crawl API call failed: %s", e)
raise
pages: List[Dict[str, Any]] = []
# Process crawl results - the crawl method returns a CrawlJob object with data attribute
data_list = []
# The crawl_result is a CrawlJob object with a 'data' attribute containing list of Document objects
if hasattr(crawl_result, 'data'):
data_list = crawl_result.data if crawl_result.data else []
logger.info("Status: %s", getattr(crawl_result, 'status', 'unknown'))
logger.info("Retrieved %d pages", len(data_list))
# Debug: Check other attributes if no data
if not data_list:
logger.debug("CrawlJob attributes: %s", [attr for attr in dir(crawl_result) if not attr.startswith('_')])
logger.debug("Status: %s", getattr(crawl_result, 'status', 'N/A'))
logger.debug("Total: %s", getattr(crawl_result, 'total', 'N/A'))
logger.debug("Completed: %s", getattr(crawl_result, 'completed', 'N/A'))
elif isinstance(crawl_result, dict) and 'data' in crawl_result:
data_list = crawl_result.get("data", [])
else:
logger.warning("Unexpected crawl result type")
logger.debug("Result type: %s", type(crawl_result))
if hasattr(crawl_result, '__dict__'):
logger.debug("Result attributes: %s", list(crawl_result.__dict__.keys()))
for item in data_list:
# Process each crawled page - properly handle object serialization
page_url = "Unknown URL"
title = ""
content_markdown = None
content_html = None
metadata = {}
# Extract data from the item
if hasattr(item, 'model_dump'):
# Pydantic model - use model_dump to get dict
item_dict = item.model_dump()
content_markdown = item_dict.get('markdown')
content_html = item_dict.get('html')
metadata = item_dict.get('metadata', {})
elif hasattr(item, '__dict__'):
# Regular object with attributes
content_markdown = getattr(item, 'markdown', None)
content_html = getattr(item, 'html', None)
# Handle metadata - convert to dict if it's an object
metadata_obj = getattr(item, 'metadata', {})
if hasattr(metadata_obj, 'model_dump'):
metadata = metadata_obj.model_dump()
elif hasattr(metadata_obj, '__dict__'):
metadata = metadata_obj.__dict__
elif isinstance(metadata_obj, dict):
metadata = metadata_obj
else:
metadata = {}
elif isinstance(item, dict):
# Already a dictionary
content_markdown = item.get('markdown')
content_html = item.get('html')
metadata = item.get('metadata', {})
# Ensure metadata is a dict (not an object)
if not isinstance(metadata, dict):
if hasattr(metadata, 'model_dump'):
metadata = metadata.model_dump()
elif hasattr(metadata, '__dict__'):
metadata = metadata.__dict__
else:
metadata = {}
# Extract URL and title from metadata
page_url = metadata.get("sourceURL", metadata.get("url", "Unknown URL"))
title = metadata.get("title", "")
# Re-check crawled page URL against policy
page_blocked = check_website_access(page_url)
if page_blocked:
logger.info("Blocked crawled page %s by rule %s", page_blocked["host"], page_blocked["rule"])
pages.append({
"url": page_url, "title": title, "content": "", "raw_content": "",
"error": page_blocked["message"],
"blocked_by_policy": {"host": page_blocked["host"], "rule": page_blocked["rule"], "source": page_blocked["source"]},
})
continue
# Choose content (prefer markdown)
content = content_markdown or content_html or ""
pages.append({
"url": page_url,
"title": title,
"content": content,
"raw_content": content,
"metadata": metadata # Now guaranteed to be a dict
})
response = {"results": pages}
pages_crawled = len(response.get('results', []))
logger.info("Crawled %d pages", pages_crawled)
debug_call_data["pages_crawled"] = pages_crawled
debug_call_data["original_response_size"] = len(json.dumps(response))
# Process each result with LLM if enabled
if use_llm_processing and auxiliary_available:
logger.info("Processing crawled content with LLM (parallel)...")
debug_call_data["processing_applied"].append("llm_processing")
# Prepare tasks for parallel processing
async def process_single_crawl_result(result):
"""Process a single crawl result with LLM and return updated result with metrics."""
page_url = result.get('url', 'Unknown URL')
title = result.get('title', '')
content = result.get('content', '')
if not content:
return result, None, "no_content"
original_size = len(content)
# Process content with LLM
processed = await process_content_with_llm(
content, page_url, title, effective_model, min_length
)
if processed:
processed_size = len(processed)
compression_ratio = processed_size / original_size if original_size > 0 else 1.0
# Update result with processed content
result['raw_content'] = content
result['content'] = processed
metrics = {
"url": page_url,
"original_size": original_size,
"processed_size": processed_size,
"compression_ratio": compression_ratio,
"model_used": effective_model
}
return result, metrics, "processed"
else:
metrics = {
"url": page_url,
"original_size": original_size,
"processed_size": original_size,
"compression_ratio": 1.0,
"model_used": None,
"reason": "content_too_short"
}
return result, metrics, "too_short"
# Run all LLM processing in parallel
results_list = response.get('results', [])
tasks = [process_single_crawl_result(result) for result in results_list]
processed_results = await asyncio.gather(*tasks)
# Collect metrics and print results
for result, metrics, status in processed_results:
page_url = result.get('url', 'Unknown URL')
if status == "processed":
debug_call_data["compression_metrics"].append(metrics)
debug_call_data["pages_processed_with_llm"] += 1
logger.info("%s (processed)", page_url)
elif status == "too_short":
debug_call_data["compression_metrics"].append(metrics)
logger.info("%s (no processing - content too short)", page_url)
else:
logger.warning("%s (no content to process)", page_url)
else:
if use_llm_processing and not auxiliary_available:
logger.warning("LLM processing requested but no auxiliary model available, returning raw content")
debug_call_data["processing_applied"].append("llm_processing_unavailable")
# Print summary of crawled pages for debugging (original behavior)
for result in response.get('results', []):
page_url = result.get('url', 'Unknown URL')
content_length = len(result.get('content', ''))
logger.info("%s (%d characters)", page_url, content_length)
# Trim output to minimal fields per entry: title, content, error
trimmed_results = [
# No registered provider supports crawl AND no crawl-capable plugin
# is available. Surface a typed error pointing the user at the two
# crawl-capable providers (Firecrawl + Tavily).
return json.dumps(
{
"url": r.get("url", ""),
"title": r.get("title", ""),
"content": r.get("content", ""),
"error": r.get("error"),
**({ "blocked_by_policy": r["blocked_by_policy"]} if "blocked_by_policy" in r else {}),
}
for r in response.get("results", [])
]
trimmed_response = {"results": trimmed_results}
result_json = json.dumps(trimmed_response, indent=2, ensure_ascii=False)
# Clean base64 images from crawled content
cleaned_result = clean_base64_images(result_json)
debug_call_data["final_response_size"] = len(cleaned_result)
debug_call_data["processing_applied"].append("base64_image_removal")
# Log debug information
_debug.log_call("web_crawl_tool", debug_call_data)
_debug.save()
return cleaned_result
"success": False,
"error": (
"web_crawl has no available backend. "
"Set FIRECRAWL_API_KEY (or FIRECRAWL_API_URL for "
f"self-hosted){_firecrawl_backend_help_suffix()}, "
"or set TAVILY_API_KEY for Tavily. "
"Alternatively use web_search + web_extract instead."
),
},
ensure_ascii=False,
)
except Exception as e:
error_msg = f"Error crawling website: {str(e)}"
logger.debug("%s", error_msg)