mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-18 04:41:56 +00:00
9d42c2c286
* feat(video_gen): unified video_generate tool with pluggable provider backends One core video_generate tool, every backend a plugin. Mirrors the image_gen + memory_provider + context_engine architecture: ABC, registry, plugin-context registration hook, and per-plugin model catalogs surfaced through hermes tools. Surface (one schema, every backend): - operation: generate / edit / extend - modalities: text-to-video (prompt only), image-to-video (prompt + image_url), video edit (prompt + video_url), video extend (video_url) - reference_image_urls, duration, aspect_ratio, resolution, negative_prompt, audio, seed, model override - Providers ignore unknown kwargs and declare what they support via VideoGenProvider.capabilities() — backend-specific quirks stay in the backend, the agent learns one tool Backends shipped: - plugins/video_gen/xai/ — Grok-Imagine, full generate/edit/extend + image-to-video + reference images (salvaged from PR #10600 by @Jaaneek, reshaped into the plugin interface) - plugins/video_gen/fal/ — Veo 3.1 (t2v + i2v), Kling O3 i2v, Pixverse v6 i2v with model-aware payload building that drops keys a model doesn't declare Wiring: - agent/video_gen_provider.py — VideoGenProvider ABC, normalize_operation, success_response / error_response, save_b64_video / save_bytes_video, $HERMES_HOME/cache/videos/ - agent/video_gen_registry.py — thread-safe register/get/list + get_active_provider() reading video_gen.provider from config.yaml - hermes_cli/plugins.py — PluginContext.register_video_gen_provider() - hermes_cli/tools_config.py — Video Generation category in hermes tools, plugin-only providers list, model picker per plugin, config write to video_gen.{provider,model} - toolsets.py — new video_gen toolset - tests: 31 new tests covering ABC, registry, tool dispatch, both plugins - docs: developer-guide/video-gen-provider-plugin.md (parallel to the image-gen guide), sidebar + toolsets-reference + plugin guides updated Supersedes: #25035 (FAL), #17972 (FAL), #14543 (xAI), #13847 (HappyHorse), #10458 (provider categories), #10786 (xAI media+search bundle), #2984 (FAL duplicate), #19086 (Google Veo standalone — easy port to plugin interface). Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com> * feat(video_gen): dynamic schema reflects active backend's capabilities Address the 'capability variance' question — instead of one tool with a static schema that lies about what every backend supports, the video_generate tool now rebuilds its description at get_definitions() time based on the configured video_gen.provider and video_gen.model. The agent sees backend-specific guidance up-front: - 'fal-ai/veo3.1/image-to-video': 'image-to-video only — image_url is REQUIRED; text-only prompts will be rejected' - 'fal-ai/veo3.1' (t2v): no image_url restriction shown - xAI grok-imagine-video: 'operations: generate, edit, extend; up to 7 reference_image_urls' - Backends without edit/extend: 'not supported on this backend — surface that they need to switch backends via hermes tools' This is the same pattern PR #22694 used for delegate_task self-capping — documented in the dynamic-tool-schemas skill. Cache invalidation is free: get_tool_definitions() already memoizes on config.yaml mtime, so a mid-session backend swap rebuilds the schema automatically. Tested: - Empirical FAL OpenAPI schema check confirms image-to-video models require image_url (FAL returns HTTP 422 otherwise) — client-side rejection in FALVideoGenProvider.generate() now prevents the wasted round-trip - Live E2E: fal-ai/veo3.1/image-to-video + prompt-only → clean missing_image_url error; fal-ai/veo3.1 + prompt-only → dispatches - 6 new tests cover the builder (no config / image-only / full-surface / text-only / unknown provider / registry wiring), all passing - 37/37 in the slice, 134/134 in the broader regression set * test(video_gen/xai): full surface integration tests + cleaner schema Verified end-to-end that the xAI plugin handles every documented mode from PR #10600's surface: text-to-video, image-to-video, reference-images-to-video, video edit, video extend (with and without prompt). All five modes route to the correct xAI endpoint (/videos/generations, /videos/edits, /videos/extensions) with the right payload shape (image / reference_images / video keys), and all five client-side rejections fire before the network: edit-without-prompt, extend-without-video_url, image+refs conflict, >7 references, and duration/aspect_ratio clamping. 15 new integration tests grouped into four classes (endpoint routing, modalities, validation, clamping). httpx is stubbed via a small fake AsyncClient that records POSTs so the tests assert the actual payload the plugin would send to xAI — not just the success/error envelope. Also cleaned up a description redundancy: when a model's operations match the backend's overall set, we no longer print the duplicate 'operations supported by this model' line. xAI's description now reads: Active backend: xAI . model: grok-imagine-video - operations supported by this backend: edit, extend, generate - modalities supported by this backend: image, reference_images, text - aspect_ratio choices: 16:9, 1:1, 2:3, 3:2, 3:4, 4:3, 9:16 - resolution choices: 480p, 720p - duration range: 1-15s - reference_image_urls: up to 7 images Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com> * feat(video_gen): collapse surface to t2v + i2v, family-based auto-routing Two design changes per Teknium: 1) Drop edit/extend from the tool surface entirely. Only text-to-video and image-to-video remain. The agent sees a clean tool with two modalities; backend-specific quirks like xAI's edit/extend endpoints stay out of the unified schema. 2) FAL: pick a model FAMILY once, the plugin routes between the family's text-to-video and image-to-video endpoints based on whether image_url was passed. Users no longer pick 'fal-ai/veo3.1' AND 'fal-ai/veo3.1/image-to-video' as separate options — they pick 'veo3.1', and the plugin handles the rest. Catalog rewritten as families: veo3.1 fal-ai/veo3.1 / fal-ai/veo3.1/image-to-video pixverse-v6 fal-ai/pixverse/v6/text-to-video / fal-ai/pixverse/v6/image-to-video kling-o3-standard fal-ai/kling-video/o3/standard/text-to-video / fal-ai/kling-video/o3/standard/image-to-video xAI uses a single endpoint (/videos/generations) for both modes, routed by the presence of the 'image' field in the payload — no edit/extend exposure. Schema changes: - VIDEO_GENERATE_SCHEMA: drop operation, drop video_url. Final params: prompt (required), image_url, reference_image_urls, duration, aspect_ratio, resolution, negative_prompt, audio, seed, model. - VideoGenProvider ABC: drop normalize_operation, VALID_OPERATIONS, DEFAULT_OPERATION. capabilities() drops 'operations' key. - success_response: add 'modality' field ('text' | 'image') so the agent and logs can see which endpoint was actually hit. Dynamic schema builder simplified — no operations bullet, no 'switch backends if you need edit/extend' guidance. When the active backend supports both modalities (the common case), description reads: Active backend: FAL . model: pixverse-v6 - supports both text-to-video (omit image_url) and image-to-video (pass image_url) - routes automatically - aspect_ratio choices: 16:9, 9:16, 1:1 - resolution choices: 360p, 540p, 720p, 1080p - duration range: 1-15s - audio: pass audio=true to enable native audio (pricing tier) - negative_prompt: supported Tests: 51 in the video_gen slice, 216 across the broader image+video sweep, all passing. New FAL routing tests prove pixverse-v6 + no image hits text-to-video endpoint, pixverse-v6 + image_url hits image-to-video endpoint, same for veo3.1 and kling-o3-standard. Docs updated: developer-guide page rewrites the 'model families' pattern as a first-class section so external plugin authors know the convention. toolsets-reference and toolsets.py descriptions match the new surface. Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com> * feat(video_gen/fal): expand catalog to 6 families, cheap + premium tiers Catalog now covers everything Teknium specced from FAL: Cheap tier: ltx-2.3 fal-ai/ltx-2.3-22b/text-to-video / image-to-video pixverse-v6 fal-ai/pixverse/v6/text-to-video / image-to-video Premium tier: veo3.1 fal-ai/veo3.1 / fal-ai/veo3.1/image-to-video seedance-2.0 bytedance/seedance-2.0/text-to-video / image-to-video kling-v3-4k fal-ai/kling-video/v3/4k/text-to-video / image-to-video happy-horse fal-ai/happy-horse/text-to-video / image-to-video DEFAULT_MODEL moved from veo3.1 (premium) to pixverse-v6 (cheap, sane defaults, both modalities) — better first-run UX for users who haven't explicitly picked a model. New family-entry knob: image_param_key. Kling v3 4K's image-to-video endpoint expects start_image_url instead of image_url; declaring image_param_key='start_image_url' on the family lets _build_payload remap correctly. Other families default to plain image_url. Per-family capability flags reflect each model's docs: - LTX 2.3 + Happy Horse: minimal payloads (no duration/aspect/resolution enum exposed by FAL — let endpoint apply defaults) - Seedance: 6 aspect ratios incl 21:9, durations 4-15, audio supported, negative prompts NOT supported per docs - Kling v3 4K: 16:9/9:16/1:1, 3-15s, audio + negative - Veo 3.1: unchanged, 16:9/9:16, 4/6/8s Tests: +5 covering the new families (full catalog, Kling 4K start_image_url remap, Seedance routing, LTX payload minimality, Happy Horse minimality). 56/56 in the slice green. Note: I did NOT add the FAL-hosted xAI Grok-Imagine variant. Hermes already has a direct xAI plugin that talks to xAI's own API; routing the same model through FAL's wrapper would duplicate the surface without adding capabilities. Users on FAL who want Grok-Imagine should use the xAI plugin directly; flag if you want both routes available. * test(video_gen): tool-surface routing matrix — every model x modality End-to-end matrix test driven through _handle_video_generate() — the actual function the agent's video_generate tool call lands in. Writes config.yaml, invokes the registered handler with a raw args dict, then asserts the outbound HTTP/SDK call hit the right endpoint with the right payload shape. Parametrized over FAL_FAMILIES.keys() so the matrix auto-discovers new families as they're added (add a family to FAL_FAMILIES and you get both modalities tested for free). Coverage: - All 6 FAL families x {text-only, text+image} = 12 cases - xAI x {text-only, text+image} = 2 cases - tool-level model= arg overrides config = 2 cases For each case, verifies: - result['success'] is True - result['modality'] matches input shape ('text' if no image_url, 'image' otherwise) - outbound endpoint URL matches the family's text_endpoint or image_endpoint - text-only payloads carry no image-shaped keys - text+image payloads carry the family's image key (image_url for most, start_image_url for kling-v3-4k, wrapped 'image' object for xAI) All 16 cases passing. Confirms the tool surface routes every (provider, model, modality) combination correctly with zero leakage. * feat(video_gen): keep video_gen out of first-run setup, surface in status Two changes: 1. video_gen joins _DEFAULT_OFF_TOOLSETS, so it is NOT pre-selected in the first-run toolset checklist. Video gen is niche, paid, and slow — most users don't want it nagging them during initial setup. Anyone who wants it opts in via 'hermes tools' -> Video Generation, which already routes to the provider+model picker. 2. The 'hermes setup' status panel learns about video_gen — but only shows the row when a plugin reports available. Users without FAL_KEY/XAI_API_KEY see nothing about video gen; users with one of those keys see 'Video Generation (FAL) ✓' as confirmation it's wired. Verified live: - Fresh install (no creds): zero video_gen mentions in wizard. - With FAL_KEY: status row appears with active backend name. - 160/160 in the setup + tools_config + video_gen test slice. Rationale: image_gen is on by default because it's a featured creative tool used in casual chat (telegrams, etc). Video gen is heavier — long wait, paid per-second pricing. Default-off matches user intent better. --------- Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
265 lines
8 KiB
TypeScript
265 lines
8 KiB
TypeScript
import type {SidebarsConfig} from '@docusaurus/plugin-content-docs';
|
|
|
|
const sidebars: SidebarsConfig = {
|
|
docs: [
|
|
'user-stories',
|
|
{
|
|
type: 'category',
|
|
label: 'Getting Started',
|
|
collapsed: true,
|
|
items: [
|
|
'getting-started/quickstart',
|
|
'getting-started/installation',
|
|
'getting-started/termux',
|
|
'getting-started/nix-setup',
|
|
'getting-started/updating',
|
|
'getting-started/learning-path',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Using Hermes',
|
|
collapsed: true,
|
|
items: [
|
|
'user-guide/cli',
|
|
'user-guide/tui',
|
|
'user-guide/windows-native',
|
|
'user-guide/windows-wsl-quickstart',
|
|
'user-guide/configuration',
|
|
'user-guide/configuring-models',
|
|
'user-guide/sessions',
|
|
'user-guide/profiles',
|
|
'user-guide/profile-distributions',
|
|
'user-guide/git-worktrees',
|
|
'user-guide/docker',
|
|
'user-guide/security',
|
|
'user-guide/checkpoints-and-rollback',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Features',
|
|
collapsed: true,
|
|
items: [
|
|
'user-guide/features/overview',
|
|
'user-guide/features/tool-gateway',
|
|
{
|
|
type: 'category',
|
|
label: 'Core',
|
|
items: [
|
|
'user-guide/features/tools',
|
|
'user-guide/features/skills',
|
|
'user-guide/features/lsp',
|
|
'user-guide/features/curator',
|
|
'user-guide/features/memory',
|
|
'user-guide/features/memory-providers',
|
|
'user-guide/features/context-files',
|
|
'user-guide/features/context-references',
|
|
'user-guide/features/personality',
|
|
'user-guide/features/skins',
|
|
'user-guide/features/plugins',
|
|
'user-guide/features/built-in-plugins',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Automation',
|
|
items: [
|
|
'user-guide/features/cron',
|
|
'user-guide/features/delegation',
|
|
'user-guide/features/kanban',
|
|
'user-guide/features/kanban-tutorial',
|
|
'user-guide/features/kanban-worker-lanes',
|
|
'user-guide/features/goals',
|
|
'user-guide/features/code-execution',
|
|
'user-guide/features/hooks',
|
|
'user-guide/features/batch-processing',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Media & Web',
|
|
items: [
|
|
'user-guide/features/voice-mode',
|
|
'user-guide/features/web-search',
|
|
'user-guide/features/browser',
|
|
'user-guide/features/computer-use',
|
|
'user-guide/features/vision',
|
|
'user-guide/features/image-generation',
|
|
'user-guide/features/tts',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Management',
|
|
items: [
|
|
'user-guide/features/web-dashboard',
|
|
'user-guide/features/extending-the-dashboard',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Advanced',
|
|
items: [
|
|
'user-guide/features/rl-training',
|
|
'user-guide/features/spotify',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Skills',
|
|
collapsed: true,
|
|
items: [
|
|
'reference/skills-catalog',
|
|
'reference/optional-skills-catalog',
|
|
],
|
|
},
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Messaging Platforms',
|
|
collapsed: true,
|
|
items: [
|
|
'user-guide/messaging/index',
|
|
'user-guide/messaging/telegram',
|
|
'user-guide/messaging/discord',
|
|
'user-guide/messaging/slack',
|
|
'user-guide/messaging/whatsapp',
|
|
'user-guide/messaging/signal',
|
|
'user-guide/messaging/email',
|
|
'user-guide/messaging/sms',
|
|
'user-guide/messaging/homeassistant',
|
|
'user-guide/messaging/mattermost',
|
|
'user-guide/messaging/matrix',
|
|
'user-guide/messaging/dingtalk',
|
|
'user-guide/messaging/feishu',
|
|
'user-guide/messaging/wecom',
|
|
'user-guide/messaging/wecom-callback',
|
|
'user-guide/messaging/weixin',
|
|
'user-guide/messaging/bluebubbles',
|
|
'user-guide/messaging/qqbot',
|
|
'user-guide/messaging/yuanbao',
|
|
'user-guide/messaging/teams',
|
|
'user-guide/messaging/teams-meetings',
|
|
'user-guide/messaging/msgraph-webhook',
|
|
'user-guide/messaging/line',
|
|
'user-guide/messaging/open-webui',
|
|
'user-guide/messaging/webhooks',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Integrations',
|
|
collapsed: true,
|
|
items: [
|
|
'integrations/index',
|
|
'integrations/providers',
|
|
'user-guide/features/mcp',
|
|
'user-guide/features/acp',
|
|
'user-guide/features/api-server',
|
|
'user-guide/features/honcho',
|
|
'user-guide/features/provider-routing',
|
|
'user-guide/features/fallback-providers',
|
|
'user-guide/features/credential-pools',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Guides & Tutorials',
|
|
collapsed: true,
|
|
items: [
|
|
'guides/tips',
|
|
'guides/local-llm-on-mac',
|
|
'guides/daily-briefing-bot',
|
|
'guides/team-telegram-assistant',
|
|
'guides/python-library',
|
|
'guides/use-mcp-with-hermes',
|
|
'guides/use-soul-with-hermes',
|
|
'guides/use-voice-mode-with-hermes',
|
|
'guides/build-a-hermes-plugin',
|
|
'guides/automate-with-cron',
|
|
'guides/cron-script-only',
|
|
'guides/automation-templates',
|
|
'guides/cron-troubleshooting',
|
|
'guides/work-with-skills',
|
|
'guides/delegation-patterns',
|
|
'guides/github-pr-review-agent',
|
|
'guides/webhook-github-pr-review',
|
|
'guides/migrate-from-openclaw',
|
|
'guides/aws-bedrock',
|
|
'guides/azure-foundry',
|
|
'guides/microsoft-graph-app-registration',
|
|
'guides/operate-teams-meeting-pipeline',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Developer Guide',
|
|
collapsed: true,
|
|
items: [
|
|
'developer-guide/contributing',
|
|
{
|
|
type: 'category',
|
|
label: 'Architecture',
|
|
items: [
|
|
'developer-guide/architecture',
|
|
'developer-guide/agent-loop',
|
|
'developer-guide/prompt-assembly',
|
|
'developer-guide/context-compression-and-caching',
|
|
'developer-guide/gateway-internals',
|
|
'developer-guide/session-storage',
|
|
'developer-guide/provider-runtime',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Extending',
|
|
items: [
|
|
'developer-guide/adding-tools',
|
|
'developer-guide/adding-providers',
|
|
'developer-guide/adding-platform-adapters',
|
|
'developer-guide/memory-provider-plugin',
|
|
'developer-guide/context-engine-plugin',
|
|
'developer-guide/model-provider-plugin',
|
|
'developer-guide/image-gen-provider-plugin',
|
|
'developer-guide/video-gen-provider-plugin',
|
|
'developer-guide/plugin-llm-access',
|
|
'developer-guide/creating-skills',
|
|
'developer-guide/extending-the-cli',
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Internals',
|
|
items: [
|
|
'developer-guide/tools-runtime',
|
|
'developer-guide/acp-internals',
|
|
'developer-guide/cron-internals',
|
|
'developer-guide/environments',
|
|
'developer-guide/trajectory-format',
|
|
],
|
|
},
|
|
],
|
|
},
|
|
{
|
|
type: 'category',
|
|
label: 'Reference',
|
|
items: [
|
|
'reference/cli-commands',
|
|
'reference/slash-commands',
|
|
'reference/profile-commands',
|
|
'reference/environment-variables',
|
|
'reference/tools-reference',
|
|
'reference/toolsets-reference',
|
|
'reference/mcp-config-reference',
|
|
'reference/model-catalog',
|
|
'reference/skills-catalog',
|
|
'reference/optional-skills-catalog',
|
|
'reference/faq',
|
|
],
|
|
},
|
|
],
|
|
};
|
|
|
|
export default sidebars;
|