2 commits

Author	SHA1	Message	Date
kshitijk4poor	21e3a863bb	feat(web): firecrawl plugin natively supports crawl; delete legacy inline path ... The web-provider migration originally left firecrawl crawl as the only provider-specific code remaining inline in tools/web_tools.py (~250 lines of Firecrawl-specific crawl orchestration that didn't fit the plugin's existing surface). This commit closes that gap. What this adds -------------- 1. plugins/web/firecrawl/provider.py: implement async ``crawl(url, **kwargs)`` - Accepts the same kwargs as the dispatcher passes to any crawl provider (``instructions``, ``depth``, ``limit``); Firecrawl's /crawl endpoint ignores ``instructions`` and ``depth`` so we log and drop with a clear info message. - Wraps the sync SDK ``crawl()`` call in asyncio.to_thread so the gateway event loop isn't blocked on a multi-page crawl. - Preserves the response-shape normalization across pydantic / typed-object / dict variants that the legacy inline code did. - Preserves per-page website-policy re-check (catches blocked redirects after the SDK returns). - Returns the same {"results": [...]} shape so the dispatcher's shared LLM-summarization post-processing path works unchanged. - Sets supports_crawl() to True so the dispatcher routes through the plugin instead of the legacy fallthrough. 2. tools/web_tools.py: delete the entire legacy firecrawl crawl block that used to run after "No registered provider supports crawl" — ~270 lines including: - check_firecrawl_api_key gate + typed error - inline SSRF + website-policy seed-URL gate (dispatcher already does this) - Firecrawl client setup with crawl_params - 100+ lines of pydantic/dict/typed-object normalization - Per-page LLM-processing loop (kept in the dispatcher's shared post-processing path; that's where it always belonged) - trimming + base64 image cleanup (still done in the dispatcher's shared path) Replaced with a single typed-error branch when no crawl-capable provider is available: "web_crawl has no available backend. Set FIRECRAWL_API_KEY (or FIRECRAWL_API_URL for self-hosted), or set TAVILY_API_KEY for Tavily." Test updates ------------ - tests/tools/test_website_policy.py: - test_web_crawl_short_circuits_blocked_url: dispatcher seed-URL gate still runs on web_tools.check_website_access (no change to that patch), but the firecrawl client lockdown moved to the plugin module — patch firecrawl_provider._get_firecrawl_client instead of web_tools._get_firecrawl_client. The dispatcher short-circuits before the plugin runs, so the test still passes. - test_web_crawl_blocks_redirected_final_url: patch the per-page policy gate at plugins.web.firecrawl.provider.check_website_access (where it now runs) AND on web_tools (where the seed-URL gate still runs). Patch firecrawl_provider._get_firecrawl_client for the FakeCrawlClient injection. Both checks flow through the same fake_check function. - tests/plugins/web/test_web_search_provider_plugins.py: - Update parametrized capability-flag spec: firecrawl supports_crawl is now True. - Add test_firecrawl_crawl_returns_error_dict_when_unconfigured — verifies inspect.iscoroutinefunction(p.crawl) is True and that the async crawl returns a per-page error dict (not a raise) when FIRECRAWL_API_KEY is missing. Verified -------- - 218/218 web tests pass (was 173, +44 plugin tests + 1 new firecrawl crawl test from this commit = 218 with the test deduplication). - Compile-clean (py_compile passes on both files). - Provider capabilities matrix confirmed end-to-end: name search extract crawl async-extract? async-crawl? firecrawl True True True True True tavily True True True False False Both crawl-capable providers exercise the dispatcher's inspect.iscoroutinefunction async-or-sync detection. Net diff -------- - tools/web_tools.py: -254 lines (legacy inline crawl gone) - plugins/web/firecrawl/provider.py: +185 lines (crawl method) - test_website_policy.py: +14/-9 lines (patch locations) - test_web_search_provider_plugins.py: +22/-1 lines (capability flag + new firecrawl crawl test) - Total: -32 net LoC; tools/web_tools.py is now 1509 lines (was 1763 before this commit, 2227 before the migration started).	2026-05-13 22:31:28 -07:00
kshitijk4poor	e8cee87e85	test(plugins): tests/plugins/web/ — coverage for the 7-plugin migration ... Adds 44 focused tests under tests/plugins/web/ covering the surface that the PR #25182 web-provider migration introduced. Complements the existing tests/tools/ coverage which is dispatcher-centric; this file is plugin-centric and tests each plugin + the registry directly. Test classes (44 tests, ~1.1s on 4 workers) ------------------------------------------- TestBundledPluginsRegister (16 tests) - All seven plugins present in the registry after _ensure_plugins_discovered() - Per-plugin parametrized capability-flag assertions (brave-free / ddgs / searxng: search-only; exa / parallel / firecrawl: search + extract; tavily: search + extract + crawl) - Every plugin exposes name + display_name properties - Every plugin returns a picker-compatible get_setup_schema() dict TestIsAvailable (7 tests) - Each premium plugin reports is_available()==False when its env var is absent and True once set (brave-free / searxng / tavily / exa / parallel) - firecrawl recognizes either FIRECRAWL_API_KEY or FIRECRAWL_API_URL as a "configured" signal - ddgs is the always-on fallback and must not raise from is_available() TestRegistryResolution (4 tests) - Option B semantics validated end-to-end: 1. Explicit configured provider wins even when is_available()==False (dispatcher surfaces typed credential errors, no silent switch) 2. Unknown/typo name falls back to first available legacy-preference provider 3. Asking for extract via a search-only backend falls back to an extract-capable available provider (capability-incompatible branch in _resolve()) 4. No config + no credentials → None (or ddgs if installed) TestAsyncExtractDispatch (4 tests) - parallel + firecrawl extract() are coroutine functions (async path in dispatcher uses await) - exa + tavily extract() are sync (dispatcher wraps in asyncio.to_thread) TestErrorResponseShapes (7 tests) - Plugins return typed error dicts (success=False + "error" key) when credentials are missing, never raise - async extract() returns list of per-URL error dicts - tavily crawl() returns {"results": [{"error": ...}]} on missing credentials Design notes ------------ - All tests use real imports of plugin modules — no mocking of provider classes themselves — so they catch drift in the ABC, registry, and glue layer simultaneously. Per the hermes-agent-dev skill's E2E testing guidance. - The autouse _isolate_env fixture clears every web-provider env var before each test so is_available() reflects the test's setup. - Resolution tests use the lower-level _resolve() directly rather than rebuilding the HERMES_HOME config dance — same observable behavior, no sys.modules.pop side-effects that would break the ABC isinstance check inside ctx.register_web_search_provider().	2026-05-13 22:31:28 -07:00