2 commits

Author	SHA1	Message	Date
kshitijk4poor	748f3e016b	refactor(web): delete inline vendor helpers, re-export from plugins ... Removes ~580 lines of dead code from tools/web_tools.py that were superseded by the plugin migration but kept around in the cutover commit to keep the diff focused. Replaces them with thin re-export shims so existing tests and external callers that reach for the legacy ``tools.web_tools.<name>`` paths continue to work transparently. Deleted from tools/web_tools.py -------------------------------- - Lazy Firecrawl SDK proxy (_load_firecrawl_cls, _FirecrawlProxy, _FIRECRAWL_CLS_CACHE, the Firecrawl singleton) - Firecrawl client section (_get_direct_firecrawl_config, _get_firecrawl_gateway_url, _is_tool_gateway_ready, _has_direct_firecrawl_config, _raise_web_backend_configuration_error, _firecrawl_backend_help_suffix, _get_firecrawl_client) - Parallel client section (_get_parallel_client, _get_async_parallel_client, _parallel_client, _async_parallel_client) - Tavily client section (_TAVILY_BASE_URL, _tavily_request, _normalize_tavily_search_results, _normalize_tavily_documents) - Generic SDK normalizers (_to_plain_object, _normalize_result_list, _extract_web_search_results, _extract_scrape_payload) - Exa client section (_get_exa_client, _exa_client, _exa_search, _exa_extract) - Parallel helpers (_parallel_search, _parallel_extract) - Duplicate inline check_firecrawl_api_key Net: tools/web_tools.py drops from 2227 → 1613 lines (-614 lines). Re-exports added at top of tools/web_tools.py --------------------------------------------- - From plugins.web.firecrawl.provider: Firecrawl, _FirecrawlProxy, _FIRECRAWL_CLS_CACHE, _load_firecrawl_cls, _get_direct_firecrawl_config, _get_firecrawl_gateway_url, _is_tool_gateway_ready, _has_direct_firecrawl_config, _firecrawl_backend_help_suffix, _raise_web_backend_configuration_error, _get_firecrawl_client, _to_plain_object, _normalize_result_list, _extract_web_search_results, _extract_scrape_payload, check_firecrawl_api_key - From plugins.web.tavily.provider: _tavily_request, _normalize_tavily_search_results, _normalize_tavily_documents - From plugins.web.parallel.provider: _get_parallel_client, _get_async_parallel_client - From plugins.web.exa.provider: _get_exa_client Plus retained module-level imports for backward-compat with tests: - httpx (tests patch tools.web_tools.httpx for tavily request mocking) - build_vendor_gateway_url, _read_nous_access_token, resolve_managed_tool_gateway, managed_nous_tools_enabled, prefers_gateway (tests patch tools.web_tools.<name>) Plugin indirection pattern (key technique) ------------------------------------------ For functions inside the firecrawl/parallel/exa plugins to honor unit-test patches that target ``tools.web_tools.<name>``, the plugin implementations now do ``import tools.web_tools as _wt`` at call time and read helper names through that module (``_wt._read_nous_access_token``, ``_wt.Firecrawl``, ``_wt.prefers_gateway``, etc.). This makes the existing test patches transparently reach the plugin code without any test changes. The cached client globals (_firecrawl_client, _firecrawl_client_config, _parallel_client, _async_parallel_client, _exa_client) also now live on tools.web_tools so existing test setup_method handlers that reset ``tools.web_tools._<vendor>_client = None`` between cases keep working. The plugins read/write the cache via getattr/setattr on the web_tools module. Verified -------- - 173/173 targeted web tests pass: test_web_providers.py, test_web_providers_brave_free.py, test_web_providers_ddgs.py, test_web_providers_searxng.py, test_web_tools_config.py, test_web_tools_tavily.py, test_website_policy.py, test_config_null_guard.py - Compile-clean (py_compile.compile passes) - All inline implementations now exist in exactly one place (plugins.web.<vendor>.provider) Follow-up clean-up ------------------ - Drop _WEB_PLUGIN_SKIPLIST + hardcoded TOOL_CATEGORIES["web"] rows (next commit) - Delete tools/web_providers/ directory entirely - Add tests/plugins/web/ coverage - Full tests/tools/ + tests/gateway/ regression sweep before promoting PR	2026-05-13 22:31:28 -07:00
kshitijk4poor	ec8449e9c6	feat(web): exa plugin — first multi-capability migration (search + extract) ... Migrates Exa from the inline `_exa_search()` / `_exa_extract()` helpers in tools/web_tools.py to a bundled plugin at plugins/web/exa/. This is the first plugin in this PR to advertise supports_extract=True, exercising the multi-capability ABC path that the initial three migrations (brave_free, ddgs, searxng — all search-only) did not cover. Both Exa methods are sync — the SDK is sync-only. The web_extract_tool dispatcher in tools/web_tools.py will continue to call them inline until Task "dispatch-extract-all" cuts it over to the registry. Behaviour preserved bit-for-bit aside from the ABC method-name change: - is_configured() -> is_available() - provider_name() -> name (property) - "exa" stays as the registered name - Module-level `_exa_client` cache + lazy `from exa_py import Exa` preserved at the new location. - Errors (ValueError for missing API key, ImportError for missing SDK, generic Exception) caught and surfaced as {"success": False, "error": ...} instead of raising. Adds "exa" to _WEB_PLUGIN_SKIPLIST in hermes_cli/tools_config.py so the hardcoded TOOL_CATEGORIES["web"] row and the plugin-injected row don't duplicate during the spike. The skip-list goes away in the cleanup phase along with the hardcoded row. The legacy inline `_exa_search` / `_exa_extract` / `_get_exa_client` / `_exa_client` in tools/web_tools.py are NOT deleted yet — the dispatcher still references them. They go away in the next dispatcher-cutover commit. E2E verified: - Plugin discovers + registers - .supports_search/.supports_extract/.supports_crawl = (True, True, False) - .get_setup_schema() returns the picker row shape - resolve(): explicit exa + EXA_API_KEY -> exa; without key -> exa (registered but unavailable, dispatcher surfaces "EXA_API_KEY not set" error)	2026-05-13 22:31:28 -07:00