mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-08 03:01:47 +00:00
f67063ba81
* feat(kanban): generic diagnostics engine for task distress signals Replaces the hallucination-specific ``warnings`` / ``RecoverySection`` surface (shipped in PR #20232) with a reusable diagnostic-rule engine that covers five distress kinds in v1 and can be extended without touching UI code. The "something's wrong with this task" signal is no longer limited to phantom card ids. Closes the follow-up from #20232 discussion. New module ---------- ``hermes_cli/kanban_diagnostics.py`` — stateless, no-side-effect rule engine. Each rule is a pure function of ``(task, events, runs, now, config) -> list[Diagnostic]``. Registry is a simple list; adding a new distress kind is one function + one import, no UI or API changes required. v1 rule set ----------- * ``hallucinated_cards`` (error) — folds the existing ``completion_blocked_hallucination`` event into the new surface. * ``prose_phantom_refs`` (warning) — folds ``suspected_hallucinated_references``. * ``repeated_spawn_failures`` (error → critical at 2x threshold) — fires when ``tasks.spawn_failures >= 3``; suggests ``hermes -p <profile> doctor`` / ``auth``. * ``repeated_crashes`` (error → critical) — fires after N consecutive ``crashed`` run outcomes with no successful completion between; suggests ``hermes kanban log <id>``. * ``stuck_in_blocked`` (warning) — fires after 24h in ``blocked`` state with no comments / unblock attempts; suggests commenting. Every diagnostic carries structured ``actions`` (reclaim, reassign, unblock, cli_hint, comment, open_docs) that render consistently in both CLI and dashboard. Suggested actions are highlighted; generic recovery actions (reclaim / reassign) are available on every kind as fallbacks. Diagnostics auto-clear when the underlying failure resolves — a clean ``completed``/``edited`` event drops hallucination diagnostics, a successful run drops crash diagnostics, a comment drops stuck-blocked diagnostics. Audit events persist; the badge goes away. API --- ``plugin_api.py``: * ``/board`` now attaches ``diagnostics`` (full list) and ``warnings`` (compact summary with ``highest_severity``) per task. * ``/tasks/{id}`` attaches diagnostics so the drawer's Diagnostics section auto-opens on flagged tasks. * NEW ``/diagnostics`` endpoint — fleet-wide listing, filterable by severity, sorted critical-first. CLI --- * NEW ``hermes kanban diagnostics [--severity X] [--task id] [--json]`` — fleet view or single-task view, matches dashboard rule output so CLI users see the same picture. * ``hermes kanban show <id>`` now renders a Diagnostics section near the top with severity markers + suggested actions. Dashboard --------- * Card badge is severity-coloured (⚠ amber warning, !! orange error, !!! red critical) using ``warnings.highest_severity``. * Attention strip above the toolbar counts EVERY task with active diagnostics (not just hallucinations), severity-coloured, lists affected tasks with Open buttons when expanded. * Drawer's old ``RecoverySection`` replaced with generic ``DiagnosticsSection`` rendering a card per active diagnostic: title + detail + structured data (task-id chips when payload keys look like id lists) + action buttons. Reassign profile picker is inline per-diagnostic. Clipboard fallback uses ``.catch()`` for environments where writeText rejects. * Three-rung severity palette; amber for warning, orange for error, red for critical. Uses CSS variables so theming is straightforward. Tests ----- * NEW ``tests/hermes_cli/test_kanban_diagnostics.py`` — 14 unit tests covering each rule's positive/negative/threshold paths, severity sorting, broken-rule isolation, and sqlite3.Row integration. * Dashboard plugin tests extended: ``/diagnostics`` endpoint (empty, populated, severity-filtered), ``/board`` exposes both diagnostic list and compact summary with ``highest_severity``. * Existing hallucination-specific test (``test_board_surfaces_ warnings_field_for_hallucinated_completions``) updated to reflect the new contract: warning summary keys by diagnostic kind (``hallucinated_cards``) not event kind. 379 kanban-suite tests pass (+16 net from this PR). Live verification ----------------- Seeded all 5 diagnostic kinds + one clean + one plain-running task (7 total) into an isolated HERMES_HOME, spun up the dashboard, and verified: * Attention strip: shows ``!! 5 tasks need attention`` in the error-severity orange; Show expands to a list of 5 rows ordered critical > error > warning. * Card badges: error tasks render ``!!`` orange, warning tasks render ``⚠`` amber, clean and plain-running tasks render no badge. * Each of the 5 rules opens a correctly-coloured, correctly-styled diagnostic card in the drawer with its specific suggested action. * Live reassign from a diagnostic card flipped ``broken-ml-worker → alice`` and the drawer refreshed with the new assignee + the same diagnostic still firing (correct: spawn_failures counter hasn't reset yet). * CLI ``hermes kanban diagnostics`` prints all 5 in severity order; ``--severity error`` narrows to 3; ``kanban show <id>`` includes the Diagnostics block at the top with suggested action hint. Migration note -------------- The old ``warnings`` shape (``{count, kinds, latest_at}``) is preserved on the API but ``kinds`` now keys by diagnostic kind (``hallucinated_cards``) instead of event kind (``completion_blocked_hallucination``). ``highest_severity`` is a new required field. The dashboard was the only consumer and has been updated in the same commit; external API consumers of the ``warnings`` field will need to update their kind-match logic. * feat(kanban/diagnostics): lead titles with the actual error text The generic 'Worker crashed N runs in a row' / 'Worker failed to spawn N times' titles buried the actual cause in the data section. Operators had to open logs or expand the diagnostic to see WHY the worker is stuck — rate-limit vs insufficient quota vs bad auth vs context overflow vs network blip all looked identical at a glance. New titles: Agent crashed 3x: openai: 429 Too Many Requests - rate limit reached Agent crashed 3x: anthropic: 402 insufficient_quota - credit balance Agent crashed 3x: provider auth error: 401 Unauthorized Agent spawn failed 4x: insufficient_quota: You exceeded your current Detail keeps the full error snippet (capped at 500 chars + ellipsis for tracebacks). Title takes the first line capped at 160 chars. Fallback title if no error recorded stays honest ('no error recorded'). Tests: 4 new cases covering 429/billing/spawn/truncation. 383 total pass (+4). Live-verified on dashboard with 6 seeded scenarios (rate-limit, billing, auth, context, network, spawn-billing) — each card title leads with the actionable error text. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| conftest.py | ||
| test_ai_gateway_models.py | ||
| test_anthropic_model_flow_stale_oauth.py | ||
| test_anthropic_oauth_flow.py | ||
| test_anthropic_provider_persistence.py | ||
| test_api_key_providers.py | ||
| test_apply_model_switch_result_context.py | ||
| test_arcee_provider.py | ||
| test_argparse_flag_propagation.py | ||
| test_at_context_completion_filter.py | ||
| test_atomic_json_write.py | ||
| test_atomic_yaml_write.py | ||
| test_auth_codex_provider.py | ||
| test_auth_commands.py | ||
| test_auth_nous_provider.py | ||
| test_auth_provider_gate.py | ||
| test_auth_qwen_provider.py | ||
| test_auth_ssl_macos.py | ||
| test_aux_config.py | ||
| test_azure_detect.py | ||
| test_backup.py | ||
| test_banner.py | ||
| test_banner_git_state.py | ||
| test_banner_skills.py | ||
| test_bedrock_model_picker.py | ||
| test_chat_skills_flag.py | ||
| test_claw.py | ||
| test_clear_stale_base_url.py | ||
| test_cmd_update.py | ||
| test_coalesce_session_args.py | ||
| test_codex_cli_model_picker.py | ||
| test_codex_models.py | ||
| test_commands.py | ||
| test_completion.py | ||
| test_config.py | ||
| test_config_drift.py | ||
| test_config_env_expansion.py | ||
| test_config_env_refs.py | ||
| test_config_validation.py | ||
| test_container_aware_cli.py | ||
| test_copilot_auth.py | ||
| test_copilot_catalog_oauth_fallback.py | ||
| test_copilot_context.py | ||
| test_copilot_in_model_list.py | ||
| test_copilot_token_exchange.py | ||
| test_cron.py | ||
| test_curator_archive_prune.py | ||
| test_curator_status.py | ||
| test_custom_provider_context_length.py | ||
| test_custom_provider_model_switch.py | ||
| test_dashboard_browser_safe_imports.py | ||
| test_dashboard_lifecycle_flags.py | ||
| test_dashboard_profiles_nav_label.py | ||
| test_debug.py | ||
| test_deprecated_cwd_warning.py | ||
| test_detect_api_mode_for_url.py | ||
| test_determine_api_mode_hostname.py | ||
| test_dingtalk_auth.py | ||
| test_discord_skill_clamp_warning.py | ||
| test_doctor.py | ||
| test_doctor_command_install.py | ||
| test_env_loader.py | ||
| test_env_sanitize_on_load.py | ||
| test_fallback_cmd.py | ||
| test_gateway.py | ||
| test_gateway_linger.py | ||
| test_gateway_runtime_health.py | ||
| test_gateway_service.py | ||
| test_gateway_wsl.py | ||
| test_gemini_free_tier_setup_block.py | ||
| test_gemini_provider.py | ||
| test_gmi_provider.py | ||
| test_goals.py | ||
| test_hooks_cli.py | ||
| test_ignore_user_config_flags.py | ||
| test_image_gen_picker.py | ||
| test_kanban_boards.py | ||
| test_kanban_cli.py | ||
| test_kanban_core_functionality.py | ||
| test_kanban_db.py | ||
| test_kanban_diagnostics.py | ||
| test_launcher.py | ||
| test_list_picker_providers.py | ||
| test_logs.py | ||
| test_managed_installs.py | ||
| test_mcp_config.py | ||
| test_mcp_reload_confirm_gate.py | ||
| test_mcp_tools_config.py | ||
| test_memory_reset.py | ||
| test_model_catalog.py | ||
| test_model_normalize.py | ||
| test_model_picker_viewport.py | ||
| test_model_provider_persistence.py | ||
| test_model_switch_context_display.py | ||
| test_model_switch_copilot_api_mode.py | ||
| test_model_switch_custom_providers.py | ||
| test_model_switch_opencode_anthropic.py | ||
| test_model_switch_variant_tags.py | ||
| test_model_validation.py | ||
| test_models.py | ||
| test_models_dev_preferred_merge.py | ||
| test_non_ascii_credential.py | ||
| test_nous_hermes_non_agentic.py | ||
| test_nous_subscription.py | ||
| test_ollama_cloud_auth.py | ||
| test_ollama_cloud_provider.py | ||
| test_openai_codex_model_validation_fallback.py | ||
| test_opencode_go_in_model_list.py | ||
| test_opencode_go_validation_fallback.py | ||
| test_overlay_slug_resolution.py | ||
| test_path_completion.py | ||
| test_pin_kanban_board_env.py | ||
| test_placeholder_usage.py | ||
| test_plugin_cli_registration.py | ||
| test_plugin_scanner_recursion.py | ||
| test_plugins.py | ||
| test_plugins_cmd.py | ||
| test_profile_export_credentials.py | ||
| test_profiles.py | ||
| test_prompt_api_key.py | ||
| test_provider_config_validation.py | ||
| test_pty_bridge.py | ||
| test_reasoning_effort_menu.py | ||
| test_redact_config_bridge.py | ||
| test_regression_16767.py | ||
| test_relaunch.py | ||
| test_resolve_last_session.py | ||
| test_runtime_provider_resolution.py | ||
| test_session_browse.py | ||
| test_sessions_delete.py | ||
| test_set_config_value.py | ||
| test_setup.py | ||
| test_setup_agent_settings.py | ||
| test_setup_hermes_script.py | ||
| test_setup_irc.py | ||
| test_setup_matrix_e2ee.py | ||
| test_setup_model_provider.py | ||
| test_setup_noninteractive.py | ||
| test_setup_ollama_cloud_force_refresh.py | ||
| test_setup_openclaw_migration.py | ||
| test_setup_prompt_menus.py | ||
| test_setup_reconfigure.py | ||
| test_skills_config.py | ||
| test_skills_hub.py | ||
| test_skills_install_flags.py | ||
| test_skills_skip_confirm.py | ||
| test_skills_subparser.py | ||
| test_skin_engine.py | ||
| test_spotify_auth.py | ||
| test_status.py | ||
| test_status_model_provider.py | ||
| test_subparser_routing_fallback.py | ||
| test_subprocess_timeouts.py | ||
| test_suppress_eio_on_interrupt.py | ||
| test_tencent_tokenhub_provider.py | ||
| test_terminal_menu_fallbacks.py | ||
| test_timeouts.py | ||
| test_tips.py | ||
| test_tool_token_estimation.py | ||
| test_tools_config.py | ||
| test_tools_disable_enable.py | ||
| test_tui_npm_install.py | ||
| test_tui_resume_flow.py | ||
| test_update_autostash.py | ||
| test_update_check.py | ||
| test_update_config_clears_custom_fields.py | ||
| test_update_gateway_restart.py | ||
| test_update_hangup_protection.py | ||
| test_update_stale_dashboard.py | ||
| test_update_yes_flag.py | ||
| test_user_providers_model_switch.py | ||
| test_voice_wrapper.py | ||
| test_web_server.py | ||
| test_web_server_host_header.py | ||
| test_web_ui_build.py | ||
| test_webhook_cli.py | ||
| test_xiaomi_provider.py | ||