mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-05-18 04:41:56 +00:00
feat(plugins): run any LLM call from inside a plugin via ctx.llm (#23194)
* feat(plugins): host-owned LLM access via ctx.llm
Plugins can now ask the host to run a one-shot chat or structured
completion against the user's active model and auth, without ever
seeing an OAuth token or API key. Closes the gap where plugins that
needed bounded structured inference (receipts, CRM extraction,
support classification) had to either bring their own provider keys
or register a tool the agent had to call.
New surface on PluginContext:
- ctx.llm.complete(messages, ...)
- ctx.llm.complete_structured(instructions, input, json_schema, ...)
- async siblings ctx.llm.acomplete / acomplete_structured
Backed by the existing auxiliary_client.call_llm pipeline — every
provider, fallback chain, vision routing, and timeout policy Hermes
already supports applies automatically.
Trust gate (fail-closed by default):
- plugins.entries.<id>.llm.allow_model_override
- plugins.entries.<id>.llm.allowed_models (allowlist; '*' = any)
- plugins.entries.<id>.llm.allow_agent_id_override
- plugins.entries.<id>.llm.allow_profile_override
Embedded model@profile shorthand goes through the same gate as
explicit profile=, so it can't bypass the auth-profile policy.
Conflicting explicit and embedded profiles fail closed.
Also lands:
- plugins/plugin-llm-example/ — reference plugin that registers
/receipt-extract, demonstrating image+text structured input,
jsonschema validation, and the trust-gate config.
- website/docs/developer-guide/plugin-llm-access.md — full API docs.
- 45 unit tests covering trust gates, JSON parsing, schema
validation, image encoding, async surface, and config loading.
Validation:
- 2628 tests pass in tests/agent/
- E2E: bundled plugin loaded with isolated HERMES_HOME, slash
command produced parsed JSON via stubbed call_llm
- response_format extra_body wired correctly for both json_object
and json_schema modes
* docs(plugin-llm): rewrite quickstart and framing
The quickstart now uses a meeting-notes-to-tasks example instead of
a receipt extractor, and the page leads with hook-time / gateway
pre-filter / scheduled-job framing rather than the OpenClaw
KB/support/CRM/finance/migration enumeration that the original
upstream PR used. Receipt example moved to a separate worked
example link so the docs page itself doesn't echo any of the
upstream framing.
Also clarifies where ctx.llm fits in the broader plugin surface
(table comparing register_tool / register_platform / register_hook
/ etc.) and what makes this lane different from auxiliary_client
internals.
No code change.
* docs(plugin-llm): reframe as any LLM call, not just structured output
The original draft leaned heavily on complete_structured() and made
the chat lane (complete() / acomplete()) feel like a footnote.
Restructure so:
- The page title and description say 'any LLM call.'
- The lead shows BOTH a plain chat call (error rewriter) AND a
structured call (triage scorer) up top.
- Quick start has two complete plugin examples — /tldr (chat) and
/paste-to-tasks (structured).
- New 'When to use which' table for choosing complete() vs
complete_structured() vs the async siblings.
- Trust-gate sections explicitly note 'all four methods,' and the
request-shaping list calls out chat-only fields (messages) and
structured-only fields (instructions, input, json_schema)
alongside each other.
- The 'Where this fits' section now says 'for any reason,
structured or not.'
The receipt-extractor reference plugin still exists under
plugins/plugin-llm-example/ — but the docs page no longer treats
it as the canonical surface example. It's now described as 'a third
worked example, this time with image input.'
No code change.
* feat(plugin-llm): split provider/model into independent explicit kwargs
The first cut accepted a single 'provider/model' slug on every method
and split it internally. That looked clean but broke under live test:
the model-override path tried to use the slug's vendor prefix as a
literal Hermes provider id, which silently switched the user off
their aggregator (e.g. plugin asks for 'openai/gpt-4o-mini' on a user
who routes through OpenRouter — host attempted to call the 'openai'
provider directly, failed because OPENAI_API_KEY wasn't set).
New shape mirrors the host's main config:
ctx.llm.complete(
messages=[...],
provider='openrouter', # gated, optional
model='openai/gpt-4o-mini', # gated, optional
profile='work', # gated, optional
...
)
Each is independently gated by its own allow_*_override flag.
Granting model-override does NOT auto-grant provider-override.
Allowlists are now per-axis (allowed_providers, allowed_models)
matched literally against whatever string the plugin sends.
Dropped 'model@profile' embedded-suffix shorthand entirely. Hermes
doesn't use that pattern anywhere else; profile= is its own kwarg.
Live E2E (against real OpenRouter via Teknium's config) confirms:
- zero-config call works
- default-deny blocks each override with a helpful error
- model-only override stays on user's active provider (the bug)
- provider+model override switches cleanly
- allowlist refuses non-listed entries
- structured output round-trip parses + schema-validates
Tests: 49 cases (up from 45); all green. Docs updated to match the
new shape, including a 'most plugins never need this section' callout
on the trust-gate config block.
* fix+cleanup(plugin-llm): real attribution, hook-mode coverage, move example out of core
Three integration fixes for the ctx.llm surface:
1. Attribution bug — result.provider and result.model now reflect
what call_llm actually used, not placeholder fallbacks ('auto',
'default'). New _resolve_attribution() helper:
- explicit overrides win (what the call targeted)
- response.model wins for the recorded model (provider
canonicalisation: 'gpt-4o' → 'gpt-4o-2024-08-06' etc.)
- falls back to _read_main_provider() / _read_main_model()
when no override is set, so audit logs reflect the user's
active main provider/model
- 'auto' / 'default' only when EVERYTHING is empty
Live verified: zero-config call now records
provider='openrouter', model='anthropic/claude-4.7-opus-20260416'
instead of provider='auto', model='default'.
2. Hook-mode coverage — TestHookMode confirms ctx.llm.complete
works from inside a registered post_tool_call callback. The
docs page promised hook integration; now there's a test that
exercises the lazy-import path through the real invoke_hook
machinery. Two cases: traceback-rewrite hook with conditional
ctx.llm.complete, and minimal hook regression for the
sync-hook + sync-llm path.
3. Reference plugin moved out of core. plugins/plugin-llm-example/
is gone from hermes-agent — it now lives in the new
NousResearch/hermes-example-plugins companion repo. The docs
page links there. Hermes' bundled plugins should be plugins
users actually run; reference / docs-companion plugins live
externally.
Test count: 56 (up from 49). Wider sweep on tests/hermes_cli/
+ tests/gateway/ + tests/tools/ + tests/agent/ shows 16770
passing; the 12 failures are all pre-existing on origin/main
(verified by stashing this branch's changes and re-running) —
kanban-boards, delegate-task, gateway-restart, tts-routing —
none touch the plugin_llm surface.
* chore(plugins): move all example plugins to companion repo
Reference / docs-companion plugins now live exclusively in
NousResearch/hermes-example-plugins, not bundled with the core repo:
- example-dashboard
- strike-freedom-cockpit
A new fourth example, plugin-llm-async-example, was added to that
repo demonstrating ctx.llm's async surface (acomplete()) with
asyncio.gather() — registers /translate <lang>: <text> which fires
forward translation + sentiment classifier in parallel, then a
back-translation for QA. Live-tested at 2.5s for three real
provider round-trips (would be ~5-6s sequential).
Docs updated:
- developer-guide/plugin-llm-access.md links both sync and async
examples in the Reference section
- user-guide/features/extending-the-dashboard.md repoints both demo
sections to the companion repo with corrected install paths
- user-guide/features/built-in-plugins.md drops the two demo rows
- AGENTS.md notes that example plugins live in the companion repo
Net: hermes-agent's plugins/ directory now contains only plugins
users actually run (memory providers, dashboard tabs that ship real
features, the disk-cleanup hook, platform adapters). All four
demo / reference plugins live externally where they can be cloned
on demand instead of inflating the core install.
This commit is contained in:
Teknium
GitHub
parent
ae4b09ce10
commit
5aa755e4e6
16 changed files with 2540 additions and 677 deletions
12
AGENTS.md
12
AGENTS.md
|
|
@ -540,10 +540,14 @@ Full authoring guide: `website/docs/developer-guide/model-provider-plugin.md`.
|
|||
|
||||
### Dashboard / context-engine / image-gen plugin directories
|
||||
|
||||
`plugins/context_engine/`, `plugins/image_gen/`, `plugins/example-dashboard/`,
|
||||
etc. follow the same pattern (ABC + orchestrator + per-plugin directory).
|
||||
Context engines plug into `agent/context_engine.py`; image-gen providers
|
||||
into `agent/image_gen_provider.py`.
|
||||
`plugins/context_engine/`, `plugins/image_gen/`, etc. follow the same
|
||||
pattern (ABC + orchestrator + per-plugin directory). Context engines
|
||||
plug into `agent/context_engine.py`; image-gen providers into
|
||||
`agent/image_gen_provider.py`. Reference / docs-companion plugins
|
||||
(`example-dashboard`, `strike-freedom-cockpit`, `plugin-llm-example`,
|
||||
`plugin-llm-async-example`) live in the
|
||||
[`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins)
|
||||
companion repo, not in this tree.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
1046
agent/plugin_llm.py
Normal file
1046
agent/plugin_llm.py
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -290,6 +290,27 @@ class PluginContext:
|
|||
def __init__(self, manifest: PluginManifest, manager: "PluginManager"):
|
||||
self.manifest = manifest
|
||||
self._manager = manager
|
||||
# Lazy-built host-owned LLM facade — see ctx.llm property below.
|
||||
self._llm: Any = None
|
||||
|
||||
# -- host-owned LLM access ----------------------------------------------
|
||||
|
||||
@property
|
||||
def llm(self) -> Any:
|
||||
"""Return the plugin's :class:`agent.plugin_llm.PluginLlm` facade.
|
||||
|
||||
Lets trusted plugins run host-owned chat or structured completions
|
||||
against the user's active model and auth without bringing their
|
||||
own provider keys. Override capability (model, agent id, auth
|
||||
profile) is fail-closed by default and gated through
|
||||
``plugins.entries.<plugin_id>.llm.*`` config keys.
|
||||
|
||||
See :mod:`agent.plugin_llm` for the full surface."""
|
||||
if self._llm is None:
|
||||
from agent.plugin_llm import PluginLlm
|
||||
plugin_id = self.manifest.key or self.manifest.name
|
||||
self._llm = PluginLlm(plugin_id=plugin_id)
|
||||
return self._llm
|
||||
|
||||
# -- tool registration --------------------------------------------------
|
||||
|
||||
|
|
|
|||
119
plugins/example-dashboard/dashboard/dist/index.js
vendored
119
plugins/example-dashboard/dashboard/dist/index.js
vendored
|
|
@ -1,119 +0,0 @@
|
|||
/**
|
||||
* Example Dashboard Plugin
|
||||
*
|
||||
* Demonstrates how to build a dashboard plugin using the Hermes Plugin SDK.
|
||||
* No build step needed — this is a plain IIFE that uses globals from the SDK.
|
||||
*/
|
||||
(function () {
|
||||
"use strict";
|
||||
|
||||
const SDK = window.__HERMES_PLUGIN_SDK__;
|
||||
const { React } = SDK;
|
||||
const { Card, CardHeader, CardTitle, CardContent, Badge, Button } = SDK.components;
|
||||
const { useState, useEffect } = SDK.hooks;
|
||||
const { cn } = SDK.utils;
|
||||
|
||||
function ExamplePage() {
|
||||
const [greeting, setGreeting] = useState(null);
|
||||
const [loading, setLoading] = useState(false);
|
||||
|
||||
function fetchGreeting() {
|
||||
setLoading(true);
|
||||
SDK.fetchJSON("/api/plugins/example/hello")
|
||||
.then(function (data) { setGreeting(data.message); })
|
||||
.catch(function () { setGreeting("(backend not available)"); })
|
||||
.finally(function () { setLoading(false); });
|
||||
}
|
||||
|
||||
return React.createElement("div", { className: "flex flex-col gap-6" },
|
||||
// Header card
|
||||
React.createElement(Card, null,
|
||||
React.createElement(CardHeader, null,
|
||||
React.createElement("div", { className: "flex items-center gap-3" },
|
||||
React.createElement(CardTitle, { className: "text-lg" }, "Example Plugin"),
|
||||
React.createElement(Badge, { variant: "outline" }, "v1.0.0"),
|
||||
),
|
||||
),
|
||||
React.createElement(CardContent, { className: "flex flex-col gap-4" },
|
||||
React.createElement("p", { className: "text-sm text-muted-foreground" },
|
||||
"This is an example dashboard plugin. It demonstrates using the Plugin SDK to build ",
|
||||
"custom tabs with React components, connect to backend API routes, and integrate with ",
|
||||
"the existing Hermes UI system.",
|
||||
),
|
||||
React.createElement("div", { className: "flex items-center gap-3" },
|
||||
React.createElement(Button, {
|
||||
onClick: fetchGreeting,
|
||||
disabled: loading,
|
||||
className: cn(
|
||||
"inline-flex items-center gap-2 border border-border bg-background/40 px-4 py-2",
|
||||
"text-sm font-courier transition-colors hover:bg-foreground/10 cursor-pointer",
|
||||
),
|
||||
}, loading ? "Loading..." : "Call Backend API"),
|
||||
greeting && React.createElement("span", {
|
||||
className: "text-sm font-courier text-muted-foreground",
|
||||
}, greeting),
|
||||
),
|
||||
),
|
||||
),
|
||||
|
||||
// Info card about the SDK
|
||||
React.createElement(Card, null,
|
||||
React.createElement(CardHeader, null,
|
||||
React.createElement(CardTitle, { className: "text-base" }, "Plugin SDK Reference"),
|
||||
),
|
||||
React.createElement(CardContent, null,
|
||||
React.createElement("div", { className: "grid gap-3 text-sm" },
|
||||
React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" },
|
||||
React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.React"),
|
||||
React.createElement("span", { className: "text-muted-foreground text-xs" }, "React instance — use instead of importing react"),
|
||||
),
|
||||
React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" },
|
||||
React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.hooks"),
|
||||
React.createElement("span", { className: "text-muted-foreground text-xs" }, "useState, useEffect, useCallback, useMemo, useRef, useContext, createContext"),
|
||||
),
|
||||
React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" },
|
||||
React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.components"),
|
||||
React.createElement("span", { className: "text-muted-foreground text-xs" }, "Card, Badge, Button, Input, Label, Select, Separator, Tabs, etc."),
|
||||
),
|
||||
React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" },
|
||||
React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.api"),
|
||||
React.createElement("span", { className: "text-muted-foreground text-xs" }, "Hermes API client — getStatus(), getSessions(), etc."),
|
||||
),
|
||||
React.createElement("div", { className: "flex flex-col gap-1 border border-border p-3" },
|
||||
React.createElement("span", { className: "font-medium" }, "window.__HERMES_PLUGIN_SDK__.utils"),
|
||||
React.createElement("span", { className: "text-muted-foreground text-xs" }, "cn(), timeAgo(), isoTimeAgo()"),
|
||||
),
|
||||
),
|
||||
),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
// Register this plugin — the dashboard picks it up automatically.
|
||||
window.__HERMES_PLUGINS__.register("example", ExamplePage);
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
// Page-scoped slot demo: inject a small banner at the top of /sessions.
|
||||
//
|
||||
// Built-in pages expose named slots (<page>:top, <page>:bottom) that
|
||||
// plugins can populate without overriding the whole route. The
|
||||
// manifest lists the slots we use in its `slots` array so the shell
|
||||
// knows to render <PluginSlot name="sessions:top" /> there.
|
||||
// ─────────────────────────────────────────────────────────────────────
|
||||
function SessionsTopBanner() {
|
||||
return React.createElement(Card, {
|
||||
className: "border-dashed",
|
||||
},
|
||||
React.createElement(CardContent, { className: "flex items-center gap-3 py-2" },
|
||||
React.createElement(Badge, { variant: "outline" }, "Example"),
|
||||
React.createElement("span", {
|
||||
className: "text-xs text-muted-foreground",
|
||||
}, "This banner was injected into the Sessions page by the example plugin via the ",
|
||||
React.createElement("code", { className: "font-courier" }, "sessions:top"),
|
||||
" slot."),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
window.__HERMES_PLUGINS__.registerSlot("example", "sessions:top", SessionsTopBanner);
|
||||
})();
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
{
|
||||
"name": "example",
|
||||
"label": "Example",
|
||||
"description": "Example dashboard plugin — demonstrates the plugin SDK",
|
||||
"icon": "Sparkles",
|
||||
"version": "1.0.0",
|
||||
"tab": {
|
||||
"path": "/example",
|
||||
"position": "after:skills"
|
||||
},
|
||||
"slots": ["sessions:top"],
|
||||
"entry": "dist/index.js",
|
||||
"api": "plugin_api.py"
|
||||
}
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
"""Example dashboard plugin — backend API routes.
|
||||
|
||||
Mounted at /api/plugins/example/ by the dashboard plugin system.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
|
||||
@router.get("/hello")
|
||||
async def hello():
|
||||
"""Simple greeting endpoint to demonstrate plugin API routes."""
|
||||
return {"message": "Hello from the example plugin!", "plugin": "example", "version": "1.0.0"}
|
||||
|
|
@ -1,70 +0,0 @@
|
|||
# Strike Freedom Cockpit — dashboard skin demo
|
||||
|
||||
Demonstrates how the dashboard skin+plugin system can be used to build a
|
||||
fully custom cockpit-style reskin without touching the core dashboard.
|
||||
|
||||
Two pieces:
|
||||
|
||||
- `theme/strike-freedom.yaml` — a dashboard theme YAML that paints the
|
||||
palette, typography, layout variant (`cockpit`), component chrome
|
||||
(notched card corners, scanlines, accent colors), and declares asset
|
||||
slots (`hero`, `crest`, `bg`).
|
||||
- `dashboard/` — a plugin that populates the `sidebar`, `header-left`,
|
||||
and `footer-right` slots reserved by the cockpit layout. The sidebar
|
||||
renders an MS-STATUS panel with segmented telemetry bars driven by
|
||||
real agent status; the header-left injects a COMPASS crest; the
|
||||
footer-right replaces the default org tagline.
|
||||
|
||||
## Install
|
||||
|
||||
1. **Theme** — copy the theme YAML into your Hermes home:
|
||||
|
||||
```
|
||||
cp theme/strike-freedom.yaml ~/.hermes/dashboard-themes/
|
||||
```
|
||||
|
||||
2. **Plugin** — the `dashboard/` directory gets auto-discovered because
|
||||
it lives under `plugins/` in the repo. On a user install, copy the
|
||||
whole plugin directory into `~/.hermes/plugins/`:
|
||||
|
||||
```
|
||||
cp -r . ~/.hermes/plugins/strike-freedom-cockpit
|
||||
```
|
||||
|
||||
3. Restart the web UI (or `GET /api/dashboard/plugins/rescan`), open it,
|
||||
pick **Strike Freedom** from the theme switcher.
|
||||
|
||||
## Customising the artwork
|
||||
|
||||
The sidebar plugin reads `--theme-asset-hero` and `--theme-asset-crest`
|
||||
from the active theme. Drop your own URLs into the theme YAML:
|
||||
|
||||
```yaml
|
||||
assets:
|
||||
hero: "/my-images/strike-freedom.png"
|
||||
crest: "/my-images/compass-crest.svg"
|
||||
bg: "/my-images/cosmic-era-bg.jpg"
|
||||
```
|
||||
|
||||
The plugin reads those at render time — no plugin code changes needed
|
||||
to swap artwork across themes.
|
||||
|
||||
## What this demo proves
|
||||
|
||||
The dashboard skin+plugin system supports (ref: `web/src/themes/types.ts`,
|
||||
`web/src/plugins/slots.ts`):
|
||||
|
||||
- Palette, typography, font URLs, density, radius — already present
|
||||
- **Asset URLs exposed as CSS vars** (bg / hero / crest / logo /
|
||||
sidebar / header + arbitrary `custom.*`)
|
||||
- **Raw `customCSS` blocks** injected as scoped `<style>` tags
|
||||
- **Per-component style overrides** (card / header / sidebar / backdrop /
|
||||
tab / progress / footer / badge / page) via CSS vars
|
||||
- **`layoutVariant`** — `standard`, `cockpit`, or `tiled`
|
||||
- **Plugin slots** — 10 named shell slots plugins can inject into
|
||||
(`backdrop`, `header-left/right/banner`, `sidebar`, `pre-main`,
|
||||
`post-main`, `footer-left/right`, `overlay`)
|
||||
- **Route overrides** — plugins can replace a built-in page entirely
|
||||
(`tab.override: "/"`) instead of just adding a tab
|
||||
- **Hidden plugins** — slot-only plugins that never show in the nav
|
||||
(`tab.hidden: true`) — as used here
|
||||
|
|
@ -1,309 +0,0 @@
|
|||
/**
|
||||
* Strike Freedom Cockpit — dashboard plugin demo.
|
||||
*
|
||||
* A slot-only plugin (manifest sets tab.hidden: true) that populates
|
||||
* three shell slots when the user has the ``strike-freedom`` theme
|
||||
* selected (or any theme that picks layoutVariant: cockpit):
|
||||
*
|
||||
* - sidebar → MS-STATUS panel: ENERGY / SHIELD / POWER bars,
|
||||
* ZGMF-X20A identity line, pilot block, hero
|
||||
* render (from --theme-asset-hero when the theme
|
||||
* provides one).
|
||||
* - header-left → COMPASS faction crest (uses --theme-asset-crest
|
||||
* if provided, falls back to a geometric SVG).
|
||||
* - footer-right → COSMIC ERA tagline that replaces the default
|
||||
* footer org line.
|
||||
*
|
||||
* The plugin demonstrates every extension point added alongside the
|
||||
* slot system: registerSlot, tab.hidden, reading theme asset CSS vars
|
||||
* from plugin code, and rendering above the built-in route content.
|
||||
*/
|
||||
(function () {
|
||||
"use strict";
|
||||
|
||||
const SDK = window.__HERMES_PLUGIN_SDK__;
|
||||
const PLUGINS = window.__HERMES_PLUGINS__;
|
||||
if (!SDK || !PLUGINS || !PLUGINS.registerSlot) {
|
||||
// Old dashboard bundle without slot support — bail silently rather
|
||||
// than breaking the page.
|
||||
return;
|
||||
}
|
||||
|
||||
const { React } = SDK;
|
||||
const { useState, useEffect } = SDK.hooks;
|
||||
const { api } = SDK;
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
/** Read a CSS custom property from :root. Empty string when unset. */
|
||||
function cssVar(name) {
|
||||
if (typeof document === "undefined") return "";
|
||||
return getComputedStyle(document.documentElement).getPropertyValue(name).trim();
|
||||
}
|
||||
|
||||
/** Segmented chip progress bar — 10 cells filled proportionally to value. */
|
||||
function TelemetryBar(props) {
|
||||
const { label, value, color } = props;
|
||||
const cells = [];
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const filled = Math.round(value / 10) > i;
|
||||
cells.push(
|
||||
React.createElement("span", {
|
||||
key: i,
|
||||
style: {
|
||||
flex: 1,
|
||||
height: 8,
|
||||
background: filled ? color : "rgba(255,255,255,0.06)",
|
||||
transition: "background 200ms",
|
||||
clipPath: "polygon(2px 0, 100% 0, calc(100% - 2px) 100%, 0 100%)",
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
return React.createElement(
|
||||
"div",
|
||||
{ style: { display: "flex", flexDirection: "column", gap: 4 } },
|
||||
React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
display: "flex",
|
||||
justifyContent: "space-between",
|
||||
fontSize: "0.65rem",
|
||||
letterSpacing: "0.12em",
|
||||
opacity: 0.75,
|
||||
},
|
||||
},
|
||||
React.createElement("span", null, label),
|
||||
React.createElement("span", { style: { color, fontWeight: 700 } }, value + "%"),
|
||||
),
|
||||
React.createElement(
|
||||
"div",
|
||||
{ style: { display: "flex", gap: 2 } },
|
||||
cells,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Sidebar: MS-STATUS panel
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
function SidebarSlot() {
|
||||
// Pull live-ish numbers from the status API so the plugin isn't just
|
||||
// a static decoration. Fall back to full bars if the API is slow /
|
||||
// unavailable.
|
||||
const [status, setStatus] = useState(null);
|
||||
useEffect(function () {
|
||||
let cancel = false;
|
||||
api.getStatus()
|
||||
.then(function (s) { if (!cancel) setStatus(s); })
|
||||
.catch(function () {});
|
||||
return function () { cancel = true; };
|
||||
}, []);
|
||||
|
||||
// Map real status signals to HUD telemetry. Energy/shield/power
|
||||
// aren't literal concepts on a software agent, so we read them from
|
||||
// adjacent signals: active sessions, gateway connected-platforms,
|
||||
// and agent-online health.
|
||||
const energy = status && status.gateway_online ? 92 : 18;
|
||||
const shield = status && status.connected_platforms
|
||||
? Math.min(100, 40 + (status.connected_platforms.length * 15))
|
||||
: 70;
|
||||
const power = status && status.active_sessions
|
||||
? Math.min(100, 55 + (status.active_sessions.length * 10))
|
||||
: 87;
|
||||
|
||||
const hero = cssVar("--theme-asset-hero");
|
||||
|
||||
return React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
padding: "1rem 0.75rem",
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
gap: "1rem",
|
||||
fontFamily: "var(--theme-font-display, sans-serif)",
|
||||
letterSpacing: "0.08em",
|
||||
textTransform: "uppercase",
|
||||
fontSize: "0.65rem",
|
||||
},
|
||||
},
|
||||
// Header line
|
||||
React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
borderBottom: "1px solid rgba(64,200,255,0.3)",
|
||||
paddingBottom: 8,
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
gap: 2,
|
||||
},
|
||||
},
|
||||
React.createElement("span", { style: { opacity: 0.6 } }, "ms status"),
|
||||
React.createElement("span", { style: { fontWeight: 700, fontSize: "0.85rem" } }, "zgmf-x20a"),
|
||||
React.createElement("span", { style: { opacity: 0.6, fontSize: "0.6rem" } }, "strike freedom"),
|
||||
),
|
||||
// Hero slot — only renders when the theme provides one.
|
||||
hero
|
||||
? React.createElement("div", {
|
||||
style: {
|
||||
width: "100%",
|
||||
aspectRatio: "3 / 4",
|
||||
backgroundImage: hero,
|
||||
backgroundSize: "contain",
|
||||
backgroundPosition: "center",
|
||||
backgroundRepeat: "no-repeat",
|
||||
opacity: 0.85,
|
||||
},
|
||||
"aria-hidden": true,
|
||||
})
|
||||
: React.createElement("div", {
|
||||
style: {
|
||||
width: "100%",
|
||||
aspectRatio: "3 / 4",
|
||||
border: "1px dashed rgba(64,200,255,0.25)",
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
justifyContent: "center",
|
||||
fontSize: "0.55rem",
|
||||
opacity: 0.4,
|
||||
},
|
||||
}, "hero slot — set assets.hero in theme"),
|
||||
// Pilot block
|
||||
React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
borderTop: "1px solid rgba(64,200,255,0.18)",
|
||||
borderBottom: "1px solid rgba(64,200,255,0.18)",
|
||||
padding: "8px 0",
|
||||
display: "flex",
|
||||
flexDirection: "column",
|
||||
gap: 2,
|
||||
},
|
||||
},
|
||||
React.createElement("span", { style: { opacity: 0.5, fontSize: "0.55rem" } }, "pilot"),
|
||||
React.createElement("span", { style: { fontWeight: 700 } }, "hermes agent"),
|
||||
React.createElement("span", { style: { opacity: 0.5, fontSize: "0.55rem" } }, "compass"),
|
||||
),
|
||||
// Telemetry bars
|
||||
React.createElement(TelemetryBar, { label: "energy", value: energy, color: "#ffce3a" }),
|
||||
React.createElement(TelemetryBar, { label: "shield", value: shield, color: "#3fd3ff" }),
|
||||
React.createElement(TelemetryBar, { label: "power", value: power, color: "#ff3a5e" }),
|
||||
// System online
|
||||
React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
marginTop: 4,
|
||||
padding: "6px 8px",
|
||||
border: "1px solid rgba(74,222,128,0.4)",
|
||||
color: "#4ade80",
|
||||
textAlign: "center",
|
||||
fontWeight: 700,
|
||||
fontSize: "0.6rem",
|
||||
},
|
||||
},
|
||||
status && status.gateway_online ? "system online" : "system offline",
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Header-left: COMPASS crest
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
function HeaderCrestSlot() {
|
||||
const crest = cssVar("--theme-asset-crest");
|
||||
const inner = crest
|
||||
? React.createElement("div", {
|
||||
style: {
|
||||
width: 28,
|
||||
height: 28,
|
||||
backgroundImage: crest,
|
||||
backgroundSize: "contain",
|
||||
backgroundPosition: "center",
|
||||
backgroundRepeat: "no-repeat",
|
||||
},
|
||||
"aria-hidden": true,
|
||||
})
|
||||
: React.createElement(
|
||||
"svg",
|
||||
{
|
||||
width: 28,
|
||||
height: 28,
|
||||
viewBox: "0 0 28 28",
|
||||
fill: "none",
|
||||
stroke: "currentColor",
|
||||
strokeWidth: 1.5,
|
||||
"aria-hidden": true,
|
||||
},
|
||||
React.createElement("path", { d: "M14 2 L26 14 L14 26 L2 14 Z" }),
|
||||
React.createElement("path", { d: "M14 8 L20 14 L14 20 L8 14 Z" }),
|
||||
React.createElement("circle", { cx: 14, cy: 14, r: 2, fill: "currentColor" }),
|
||||
);
|
||||
return React.createElement(
|
||||
"div",
|
||||
{
|
||||
style: {
|
||||
display: "flex",
|
||||
alignItems: "center",
|
||||
paddingLeft: 12,
|
||||
paddingRight: 8,
|
||||
color: "var(--color-accent, #3fd3ff)",
|
||||
},
|
||||
},
|
||||
inner,
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Footer-right: COSMIC ERA tagline
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
function FooterTaglineSlot() {
|
||||
return React.createElement(
|
||||
"span",
|
||||
{
|
||||
style: {
|
||||
fontFamily: "var(--theme-font-display, sans-serif)",
|
||||
fontSize: "0.6rem",
|
||||
letterSpacing: "0.18em",
|
||||
textTransform: "uppercase",
|
||||
opacity: 0.75,
|
||||
mixBlendMode: "plus-lighter",
|
||||
},
|
||||
},
|
||||
"compass hermes systems / cosmic era 71",
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Hidden tab placeholder — tab.hidden=true means this never renders in
|
||||
// the nav, but we still register something sensible in case someone
|
||||
// manually navigates to /strike-freedom-cockpit (e.g. via a bookmark).
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
function HiddenPage() {
|
||||
return React.createElement(
|
||||
"div",
|
||||
{ style: { padding: "2rem", opacity: 0.6, fontSize: "0.8rem" } },
|
||||
"Strike Freedom cockpit is a slot-only plugin — it populates the sidebar, header, and footer instead of showing a tab page.",
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Registration
|
||||
// ---------------------------------------------------------------------
|
||||
|
||||
const NAME = "strike-freedom-cockpit";
|
||||
PLUGINS.register(NAME, HiddenPage);
|
||||
PLUGINS.registerSlot(NAME, "sidebar", SidebarSlot);
|
||||
PLUGINS.registerSlot(NAME, "header-left", HeaderCrestSlot);
|
||||
PLUGINS.registerSlot(NAME, "footer-right", FooterTaglineSlot);
|
||||
})();
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
{
|
||||
"name": "strike-freedom-cockpit",
|
||||
"label": "Strike Freedom Cockpit",
|
||||
"description": "MS-STATUS sidebar + header crest for the Strike Freedom theme",
|
||||
"icon": "Shield",
|
||||
"version": "1.0.0",
|
||||
"tab": {
|
||||
"path": "/strike-freedom-cockpit",
|
||||
"position": "end",
|
||||
"hidden": true
|
||||
},
|
||||
"slots": ["sidebar", "header-left", "footer-right"],
|
||||
"entry": "dist/index.js"
|
||||
}
|
||||
|
|
@ -1,126 +0,0 @@
|
|||
# Strike Freedom — Hermes dashboard theme demo
|
||||
#
|
||||
# Copy this file to ~/.hermes/dashboard-themes/strike-freedom.yaml and
|
||||
# restart the web UI (or hit `/api/dashboard/plugins/rescan`). Pair with
|
||||
# the `strike-freedom-cockpit` plugin (plugins/strike-freedom-cockpit/)
|
||||
# for the full cockpit experience — this theme paints the palette,
|
||||
# chrome, and layout; the plugin supplies the MS-STATUS sidebar + header
|
||||
# crest that the cockpit layout variant reserves space for.
|
||||
#
|
||||
# Demonstrates every theme extension point added alongside the plugin
|
||||
# slot system: palette, typography, layoutVariant, assets, customCSS,
|
||||
# componentStyles, colorOverrides.
|
||||
name: strike-freedom
|
||||
label: "Strike Freedom"
|
||||
description: "Cockpit HUD — deep navy + cyan + gold accents"
|
||||
|
||||
# ------- palette (3-layer) -------
|
||||
palette:
|
||||
background: "#05091a"
|
||||
midground: "#d8f0ff"
|
||||
foreground:
|
||||
hex: "#ffffff"
|
||||
alpha: 0
|
||||
warmGlow: "rgba(255, 199, 55, 0.24)"
|
||||
noiseOpacity: 0.7
|
||||
|
||||
# ------- typography -------
|
||||
typography:
|
||||
fontSans: '"Orbitron", "Eurostile", "Bank Gothic", "Impact", sans-serif'
|
||||
fontMono: '"Share Tech Mono", "JetBrains Mono", ui-monospace, monospace'
|
||||
fontDisplay: '"Orbitron", "Eurostile", "Impact", sans-serif'
|
||||
fontUrl: "https://fonts.googleapis.com/css2?family=Orbitron:wght@400;500;600;700;800&family=Share+Tech+Mono&display=swap"
|
||||
baseSize: "14px"
|
||||
lineHeight: "1.5"
|
||||
letterSpacing: "0.04em"
|
||||
|
||||
# ------- layout -------
|
||||
layout:
|
||||
radius: "0"
|
||||
density: "compact"
|
||||
|
||||
# ``cockpit`` reserves a 260px left rail that the shell renders when the
|
||||
# user is on this theme. A paired plugin populates the rail via the
|
||||
# ``sidebar`` slot; with no plugin the rail shows a placeholder.
|
||||
layoutVariant: cockpit
|
||||
|
||||
# ------- assets -------
|
||||
# Use any URL (https, data:, /dashboard-plugins/...) or a pre-wrapped
|
||||
# ``url(...)``/``linear-gradient(...)`` expression. The shell exposes
|
||||
# each as a CSS var so plugins can read the same imagery.
|
||||
assets:
|
||||
bg: "linear-gradient(140deg, #05091a 0%, #0a1530 55%, #102048 100%)"
|
||||
# Plugin reads --theme-asset-hero / --theme-asset-crest to populate
|
||||
# its sidebar hero render + header crest. Replace these URLs with your
|
||||
# own artwork (copy files into ~/.hermes/dashboard-themes/assets/ and
|
||||
# reference them as /dashboard-themes-assets/strike-freedom/hero.png
|
||||
# once that static route is wired up — for now use inline data URLs or
|
||||
# remote URLs).
|
||||
hero: ""
|
||||
crest: ""
|
||||
|
||||
# ------- component chrome -------
|
||||
# Each bucket's props become CSS vars (--component-<bucket>-<kebab>) that
|
||||
# built-in shell components (Card, header, sidebar, backdrop) consume.
|
||||
componentStyles:
|
||||
card:
|
||||
# Notched corners on the top-left + bottom-right — classic mecha UI.
|
||||
clipPath: "polygon(12px 0, 100% 0, 100% calc(100% - 12px), calc(100% - 12px) 100%, 0 100%, 0 12px)"
|
||||
background: "linear-gradient(180deg, rgba(10, 22, 52, 0.85) 0%, rgba(5, 9, 26, 0.92) 100%)"
|
||||
boxShadow: "inset 0 0 0 1px rgba(64, 200, 255, 0.28), 0 0 18px -6px rgba(64, 200, 255, 0.4)"
|
||||
header:
|
||||
background: "linear-gradient(180deg, rgba(16, 32, 72, 0.95) 0%, rgba(5, 9, 26, 0.9) 100%)"
|
||||
sidebar:
|
||||
background: "linear-gradient(180deg, rgba(8, 18, 42, 0.88) 0%, rgba(5, 9, 26, 0.85) 100%)"
|
||||
tab:
|
||||
clipPath: "polygon(6px 0, 100% 0, calc(100% - 6px) 100%, 0 100%)"
|
||||
backdrop:
|
||||
backgroundSize: "cover"
|
||||
backgroundPosition: "center"
|
||||
fillerOpacity: "1"
|
||||
fillerBlendMode: "normal"
|
||||
|
||||
# ------- color overrides -------
|
||||
colorOverrides:
|
||||
primary: "#ffce3a"
|
||||
primaryForeground: "#05091a"
|
||||
accent: "#3fd3ff"
|
||||
accentForeground: "#05091a"
|
||||
ring: "#3fd3ff"
|
||||
success: "#4ade80"
|
||||
warning: "#ffce3a"
|
||||
destructive: "#ff3a5e"
|
||||
border: "rgba(64, 200, 255, 0.28)"
|
||||
|
||||
# ------- customCSS -------
|
||||
# Raw CSS injected as a scoped <style> tag on theme apply. Use this for
|
||||
# selector-level tweaks componentStyles can't express (pseudo-elements,
|
||||
# animations, media queries). Bounded to 32 KiB per theme.
|
||||
customCSS: |
|
||||
/* Scanline overlay — subtle, only when theme is active. */
|
||||
:root[data-layout-variant="cockpit"] body::before {
|
||||
content: "";
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
pointer-events: none;
|
||||
z-index: 100;
|
||||
background: repeating-linear-gradient(
|
||||
to bottom,
|
||||
transparent 0px,
|
||||
transparent 2px,
|
||||
rgba(64, 200, 255, 0.035) 3px,
|
||||
rgba(64, 200, 255, 0.035) 4px
|
||||
);
|
||||
mix-blend-mode: screen;
|
||||
}
|
||||
|
||||
/* Chevron pips on card corners. */
|
||||
[data-layout-variant="cockpit"] .border-border::before,
|
||||
[data-layout-variant="cockpit"] .border-border::after {
|
||||
content: "";
|
||||
position: absolute;
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border: 1px solid rgba(64, 200, 255, 0.55);
|
||||
pointer-events: none;
|
||||
}
|
||||
991
tests/agent/test_plugin_llm.py
Normal file
991
tests/agent/test_plugin_llm.py
Normal file
|
|
@ -0,0 +1,991 @@
|
|||
"""Unit tests for the plugin LLM facade (``agent.plugin_llm``).
|
||||
|
||||
These tests exercise the trust gate, JSON parsing, schema validation,
|
||||
image input encoding, and the auxiliary-client invocation contract.
|
||||
The auxiliary client itself is stubbed via ``make_plugin_llm_for_test``
|
||||
so we don't hit real providers.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from agent.plugin_llm import (
|
||||
PluginLlm,
|
||||
PluginLlmCompleteResult,
|
||||
PluginLlmImageInput,
|
||||
PluginLlmStructuredResult,
|
||||
PluginLlmTextInput,
|
||||
PluginLlmTrustError,
|
||||
_build_structured_messages,
|
||||
_check_overrides,
|
||||
_coerce_allowlist,
|
||||
_parse_structured_text,
|
||||
_strip_code_fences,
|
||||
_TrustPolicy,
|
||||
make_plugin_llm_for_test,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _fake_response(text: str, *, prompt: int = 4, completion: int = 6) -> SimpleNamespace:
|
||||
"""Build an OpenAI-shaped response with the given text + token usage."""
|
||||
return SimpleNamespace(
|
||||
choices=[
|
||||
SimpleNamespace(
|
||||
message=SimpleNamespace(content=text, role="assistant"),
|
||||
finish_reason="stop",
|
||||
)
|
||||
],
|
||||
usage=SimpleNamespace(
|
||||
prompt_tokens=prompt,
|
||||
completion_tokens=completion,
|
||||
total_tokens=prompt + completion,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _trusted_policy(plugin_id: str = "trusted-plugin", **overrides: Any) -> _TrustPolicy:
|
||||
defaults = dict(
|
||||
allow_provider_override=True,
|
||||
allowed_providers=None,
|
||||
allow_any_provider=True,
|
||||
allow_model_override=True,
|
||||
allowed_models=None,
|
||||
allow_any_model=True,
|
||||
allow_agent_id_override=True,
|
||||
allow_profile_override=True,
|
||||
)
|
||||
defaults.update(overrides)
|
||||
return _TrustPolicy(plugin_id=plugin_id, **defaults)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Trust gate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTrustGate:
|
||||
def test_default_policy_blocks_provider_override(self):
|
||||
policy = _TrustPolicy(plugin_id="locked")
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the provider"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider="anthropic",
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_default_policy_blocks_model_override(self):
|
||||
policy = _TrustPolicy(plugin_id="locked")
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the model"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model="claude-3-5-sonnet",
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_default_policy_blocks_agent_override(self):
|
||||
policy = _TrustPolicy(plugin_id="locked")
|
||||
with pytest.raises(PluginLlmTrustError, match="non-default agent id"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model=None,
|
||||
requested_agent_id="ada",
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_default_policy_blocks_profile_override(self):
|
||||
policy = _TrustPolicy(plugin_id="locked")
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the auth profile"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile="work",
|
||||
)
|
||||
|
||||
def test_overrides_independent(self):
|
||||
"""Each override is gated independently — turning on
|
||||
``allow_model_override`` does NOT also grant provider override."""
|
||||
policy = _TrustPolicy(
|
||||
plugin_id="model-only",
|
||||
allow_model_override=True,
|
||||
allow_any_model=True,
|
||||
)
|
||||
# model alone passes
|
||||
_, m, _, _ = _check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model="gpt-4o",
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
assert m == "gpt-4o"
|
||||
# provider alone is still denied
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the provider"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider="anthropic",
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_provider_allowlist_rejects_non_listed(self):
|
||||
policy = _TrustPolicy(
|
||||
plugin_id="restricted",
|
||||
allow_provider_override=True,
|
||||
allowed_providers=frozenset({"openrouter", "anthropic"}),
|
||||
allow_any_provider=False,
|
||||
)
|
||||
with pytest.raises(PluginLlmTrustError, match="not in plugins.entries"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider="openai",
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_provider_allowlist_accepts_listed_case_insensitively(self):
|
||||
policy = _TrustPolicy(
|
||||
plugin_id="restricted",
|
||||
allow_provider_override=True,
|
||||
allowed_providers=frozenset({"openrouter"}),
|
||||
allow_any_provider=False,
|
||||
)
|
||||
p, _, _, _ = _check_overrides(
|
||||
policy,
|
||||
requested_provider="OpenRouter",
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
assert p == "OpenRouter"
|
||||
|
||||
def test_model_allowlist_rejects_non_listed(self):
|
||||
policy = _TrustPolicy(
|
||||
plugin_id="restricted",
|
||||
allow_model_override=True,
|
||||
allowed_models=frozenset({"openai/gpt-4o-mini"}),
|
||||
allow_any_model=False,
|
||||
)
|
||||
with pytest.raises(PluginLlmTrustError, match="not in plugins.entries"):
|
||||
_check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model="anthropic/claude-3-opus",
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
|
||||
def test_model_allowlist_accepts_listed_case_insensitively(self):
|
||||
policy = _TrustPolicy(
|
||||
plugin_id="restricted",
|
||||
allow_model_override=True,
|
||||
allowed_models=frozenset({"openai/gpt-4o-mini"}),
|
||||
allow_any_model=False,
|
||||
)
|
||||
_, m, _, _ = _check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model="OpenAI/GPT-4o-mini",
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
assert m == "OpenAI/GPT-4o-mini"
|
||||
|
||||
def test_no_overrides_passes_through(self):
|
||||
policy = _TrustPolicy(plugin_id="locked")
|
||||
result = _check_overrides(
|
||||
policy,
|
||||
requested_provider=None,
|
||||
requested_model=None,
|
||||
requested_agent_id=None,
|
||||
requested_profile=None,
|
||||
)
|
||||
assert result == (None, None, None, None)
|
||||
|
||||
def test_all_overrides_when_fully_trusted(self):
|
||||
policy = _trusted_policy()
|
||||
result = _check_overrides(
|
||||
policy,
|
||||
requested_provider="openrouter",
|
||||
requested_model="anthropic/claude-3-5-sonnet",
|
||||
requested_agent_id="ada",
|
||||
requested_profile="work",
|
||||
)
|
||||
assert result == ("openrouter", "anthropic/claude-3-5-sonnet", "ada", "work")
|
||||
|
||||
|
||||
class TestAllowlistCoercion:
|
||||
def test_missing_yields_none(self):
|
||||
ranges, allow_any = _coerce_allowlist(None)
|
||||
assert ranges is None
|
||||
assert allow_any is False
|
||||
|
||||
def test_list_of_strings(self):
|
||||
ranges, allow_any = _coerce_allowlist(["A", "B"])
|
||||
assert ranges == frozenset({"a", "b"})
|
||||
assert allow_any is False
|
||||
|
||||
def test_star_alone_means_any(self):
|
||||
ranges, allow_any = _coerce_allowlist(["*"])
|
||||
assert ranges == frozenset()
|
||||
assert allow_any is True
|
||||
|
||||
def test_star_plus_specific_keeps_specifics(self):
|
||||
ranges, allow_any = _coerce_allowlist(["*", "openrouter"])
|
||||
assert ranges == frozenset({"openrouter"})
|
||||
assert allow_any is True
|
||||
|
||||
def test_non_list_yields_none(self):
|
||||
ranges, allow_any = _coerce_allowlist("openrouter")
|
||||
assert ranges is None
|
||||
assert allow_any is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Structured message building
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestStructuredMessageBuilding:
|
||||
def test_text_only_input(self):
|
||||
messages = _build_structured_messages(
|
||||
instructions="Extract the action items",
|
||||
inputs=[PluginLlmTextInput(text="meeting notes go here")],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
assert len(messages) == 1
|
||||
assert messages[0]["role"] == "user"
|
||||
parts = messages[0]["content"]
|
||||
assert parts[0]["type"] == "text"
|
||||
assert "Extract the action items" in parts[0]["text"]
|
||||
assert parts[1] == {"type": "text", "text": "meeting notes go here"}
|
||||
|
||||
def test_json_mode_adds_system_directive(self):
|
||||
messages = _build_structured_messages(
|
||||
instructions="Summarise",
|
||||
inputs=[PluginLlmTextInput(text="content")],
|
||||
json_mode=True,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
assert messages[0]["role"] == "system"
|
||||
assert "JSON object" in messages[0]["content"]
|
||||
|
||||
def test_schema_name_appended_to_header(self):
|
||||
messages = _build_structured_messages(
|
||||
instructions="Extract fields",
|
||||
inputs=[PluginLlmTextInput(text="data")],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name="action.items",
|
||||
system_prompt=None,
|
||||
)
|
||||
header = messages[0]["content"][0]["text"]
|
||||
assert "Schema name: action.items" in header
|
||||
|
||||
def test_image_bytes_encoded_as_data_url(self):
|
||||
png_bytes = b"\x89PNG\r\n\x1a\nfake"
|
||||
messages = _build_structured_messages(
|
||||
instructions="Read the image",
|
||||
inputs=[
|
||||
PluginLlmImageInput(data=png_bytes, mime_type="image/png"),
|
||||
PluginLlmTextInput(text="prefer printed text"),
|
||||
],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
parts = messages[0]["content"]
|
||||
assert parts[1]["type"] == "image_url"
|
||||
url = parts[1]["image_url"]["url"]
|
||||
assert url.startswith("data:image/png;base64,")
|
||||
decoded = base64.b64decode(url.split(",", 1)[1])
|
||||
assert decoded == png_bytes
|
||||
assert parts[2] == {"type": "text", "text": "prefer printed text"}
|
||||
|
||||
def test_image_url_passed_through(self):
|
||||
messages = _build_structured_messages(
|
||||
instructions="Caption this",
|
||||
inputs=[PluginLlmImageInput(url="https://example.com/cat.jpg")],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
img_part = messages[0]["content"][1]
|
||||
assert img_part["type"] == "image_url"
|
||||
assert img_part["image_url"]["url"] == "https://example.com/cat.jpg"
|
||||
|
||||
def test_dict_inputs_normalized(self):
|
||||
messages = _build_structured_messages(
|
||||
instructions="Test",
|
||||
inputs=[
|
||||
{"type": "text", "text": "hello"},
|
||||
{"type": "image", "url": "https://x.example/y.png"},
|
||||
],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
parts = messages[0]["content"]
|
||||
assert parts[1]["text"] == "hello"
|
||||
assert parts[2]["image_url"]["url"] == "https://x.example/y.png"
|
||||
|
||||
def test_invalid_input_block_rejected(self):
|
||||
with pytest.raises(ValueError, match="Unknown input block"):
|
||||
_build_structured_messages(
|
||||
instructions="Test",
|
||||
inputs=[{"type": "audio", "data": b""}],
|
||||
json_mode=False,
|
||||
json_schema=None,
|
||||
schema_name=None,
|
||||
system_prompt=None,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# JSON parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestJsonParsing:
|
||||
def test_strip_code_fences_with_json_label(self):
|
||||
assert _strip_code_fences('```json\n{"a":1}\n```') == '{"a":1}'
|
||||
|
||||
def test_strip_code_fences_without_label(self):
|
||||
assert _strip_code_fences("```\nfoo\n```") == "foo"
|
||||
|
||||
def test_strip_code_fences_no_fence(self):
|
||||
assert _strip_code_fences('{"a":1}') == '{"a":1}'
|
||||
|
||||
def test_parse_returns_text_when_not_json_mode(self):
|
||||
parsed, ct = _parse_structured_text(
|
||||
text='{"a": 1}', json_mode=False, json_schema=None
|
||||
)
|
||||
assert parsed is None
|
||||
assert ct == "text"
|
||||
|
||||
def test_parse_valid_json_with_json_mode(self):
|
||||
parsed, ct = _parse_structured_text(
|
||||
text='{"language": "French", "is_question": true}',
|
||||
json_mode=True,
|
||||
json_schema=None,
|
||||
)
|
||||
assert parsed == {"language": "French", "is_question": True}
|
||||
assert ct == "json"
|
||||
|
||||
def test_parse_strips_code_fences_before_loading(self):
|
||||
parsed, ct = _parse_structured_text(
|
||||
text='Here you go:\n```json\n{"ok": true}\n```',
|
||||
json_mode=True,
|
||||
json_schema=None,
|
||||
)
|
||||
assert parsed == {"ok": True}
|
||||
assert ct == "json"
|
||||
|
||||
def test_parse_returns_text_on_invalid_json(self):
|
||||
parsed, ct = _parse_structured_text(
|
||||
text="not even close to json",
|
||||
json_mode=True,
|
||||
json_schema=None,
|
||||
)
|
||||
assert parsed is None
|
||||
assert ct == "text"
|
||||
|
||||
def test_schema_validation_rejects_mismatch(self):
|
||||
pytest.importorskip("jsonschema")
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"language": {"type": "string"}},
|
||||
"required": ["language"],
|
||||
}
|
||||
with pytest.raises(ValueError, match="did not match schema"):
|
||||
_parse_structured_text(
|
||||
text='{"is_question": true}',
|
||||
json_mode=False,
|
||||
json_schema=schema,
|
||||
)
|
||||
|
||||
def test_schema_validation_accepts_match(self):
|
||||
pytest.importorskip("jsonschema")
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"language": {"type": "string"}},
|
||||
"required": ["language"],
|
||||
}
|
||||
parsed, ct = _parse_structured_text(
|
||||
text='{"language": "French"}',
|
||||
json_mode=False,
|
||||
json_schema=schema,
|
||||
)
|
||||
assert parsed == {"language": "French"}
|
||||
assert ct == "json"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# End-to-end facade
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPluginLlmFacade:
|
||||
def test_complete_uses_active_model_by_default(self):
|
||||
captured: dict = {}
|
||||
|
||||
def fake_caller(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return "auto", "default", _fake_response("Hello world.")
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
result = llm.complete([{"role": "user", "content": "hi"}])
|
||||
assert isinstance(result, PluginLlmCompleteResult)
|
||||
assert result.text == "Hello world."
|
||||
assert captured["provider_override"] is None
|
||||
assert captured["model_override"] is None
|
||||
assert captured["profile_override"] is None
|
||||
assert result.usage.input_tokens == 4
|
||||
assert result.usage.total_tokens == 10
|
||||
|
||||
def test_complete_rejects_provider_override_without_trust(self):
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=lambda **_: ("x", "y", _fake_response("")),
|
||||
)
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the provider"):
|
||||
llm.complete(
|
||||
[{"role": "user", "content": "hi"}],
|
||||
provider="openrouter",
|
||||
)
|
||||
|
||||
def test_complete_rejects_model_override_without_trust(self):
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=lambda **_: ("x", "y", _fake_response("")),
|
||||
)
|
||||
with pytest.raises(PluginLlmTrustError, match="cannot override the model"):
|
||||
llm.complete(
|
||||
[{"role": "user", "content": "hi"}],
|
||||
model="anthropic/claude-3-opus",
|
||||
)
|
||||
|
||||
def test_complete_passes_through_trusted_overrides(self):
|
||||
captured: dict = {}
|
||||
|
||||
def fake_caller(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return "anthropic", "claude-3-opus", _fake_response("ok")
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_trusted_policy("my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
result = llm.complete(
|
||||
[{"role": "user", "content": "hi"}],
|
||||
provider="anthropic",
|
||||
model="claude-3-opus",
|
||||
profile="work",
|
||||
agent_id="ada",
|
||||
temperature=0.0,
|
||||
max_tokens=128,
|
||||
timeout=10.0,
|
||||
purpose="extract",
|
||||
)
|
||||
# The recorded provider/model in the result come from the override,
|
||||
# since the stub caller echoed those values.
|
||||
assert result.provider == "anthropic"
|
||||
assert result.model == "claude-3-opus"
|
||||
assert captured["provider_override"] == "anthropic"
|
||||
assert captured["model_override"] == "claude-3-opus"
|
||||
assert captured["profile_override"] == "work"
|
||||
assert captured["temperature"] == 0.0
|
||||
assert captured["max_tokens"] == 128
|
||||
assert captured["timeout"] == 10.0
|
||||
|
||||
def test_complete_structured_returns_parsed_json(self):
|
||||
def fake_caller(**_kwargs):
|
||||
return "openai", "gpt-4o", _fake_response(
|
||||
'{"language": "French", "is_question": true, "confidence": 0.99}'
|
||||
)
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
result = llm.complete_structured(
|
||||
instructions="Detect language",
|
||||
input=[PluginLlmTextInput(text="Comment ça va?")],
|
||||
json_mode=True,
|
||||
)
|
||||
assert isinstance(result, PluginLlmStructuredResult)
|
||||
assert result.parsed == {
|
||||
"language": "French",
|
||||
"is_question": True,
|
||||
"confidence": 0.99,
|
||||
}
|
||||
assert result.content_type == "json"
|
||||
|
||||
def test_complete_structured_returns_text_on_unparseable_response(self):
|
||||
def fake_caller(**_kwargs):
|
||||
return "openai", "gpt-4o", _fake_response("Sorry, I can't help with that.")
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
result = llm.complete_structured(
|
||||
instructions="Detect language",
|
||||
input=[PluginLlmTextInput(text="x")],
|
||||
json_mode=True,
|
||||
)
|
||||
assert result.parsed is None
|
||||
assert result.content_type == "text"
|
||||
assert result.text.startswith("Sorry")
|
||||
|
||||
def test_complete_structured_validates_against_schema(self):
|
||||
pytest.importorskip("jsonschema")
|
||||
|
||||
def fake_caller(**_kwargs):
|
||||
return "openai", "gpt-4o", _fake_response('{"unrelated": "field"}')
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"language": {"type": "string"}},
|
||||
"required": ["language"],
|
||||
}
|
||||
with pytest.raises(ValueError, match="did not match schema"):
|
||||
llm.complete_structured(
|
||||
instructions="Detect language",
|
||||
input=[PluginLlmTextInput(text="x")],
|
||||
json_schema=schema,
|
||||
)
|
||||
|
||||
def test_complete_structured_requires_instructions(self):
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=MagicMock(),
|
||||
)
|
||||
with pytest.raises(ValueError, match="non-empty instructions"):
|
||||
llm.complete_structured(
|
||||
instructions=" ",
|
||||
input=[PluginLlmTextInput(text="x")],
|
||||
)
|
||||
|
||||
def test_complete_structured_requires_at_least_one_input(self):
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=MagicMock(),
|
||||
)
|
||||
with pytest.raises(ValueError, match="at least one input"):
|
||||
llm.complete_structured(
|
||||
instructions="Extract",
|
||||
input=[],
|
||||
)
|
||||
|
||||
def test_complete_structured_emits_response_format_extra_body(self):
|
||||
captured: dict = {}
|
||||
|
||||
def fake_caller(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return "openai", "gpt-4o", _fake_response('{"a": 1}')
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
schema = {"type": "object"}
|
||||
llm.complete_structured(
|
||||
instructions="Test",
|
||||
input=[PluginLlmTextInput(text="x")],
|
||||
json_schema=schema,
|
||||
)
|
||||
rf = captured["extra_body"]["response_format"]
|
||||
assert rf["type"] == "json_schema"
|
||||
assert rf["json_schema"]["schema"] == schema
|
||||
|
||||
def test_complete_structured_with_image_passes_image_url_part(self):
|
||||
captured: dict = {}
|
||||
|
||||
def fake_caller(**kwargs):
|
||||
captured.update(kwargs)
|
||||
return "openai", "gpt-4o", _fake_response('{"caption": "ok"}')
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
png = b"fake-bytes"
|
||||
llm.complete_structured(
|
||||
instructions="Caption this",
|
||||
input=[PluginLlmImageInput(data=png, mime_type="image/png")],
|
||||
json_mode=True,
|
||||
)
|
||||
msgs = captured["messages"]
|
||||
user_msg = next(m for m in msgs if m["role"] == "user")
|
||||
image_parts = [p for p in user_msg["content"] if p.get("type") == "image_url"]
|
||||
assert len(image_parts) == 1
|
||||
assert image_parts[0]["image_url"]["url"].startswith("data:image/png;base64,")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Async surface
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAsyncSurface:
|
||||
def test_acomplete_uses_async_caller(self):
|
||||
async def fake_async(**_kwargs):
|
||||
return "openai", "gpt-4o", _fake_response("async hello")
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
async_caller=fake_async,
|
||||
)
|
||||
|
||||
async def _run() -> PluginLlmCompleteResult:
|
||||
return await llm.acomplete([{"role": "user", "content": "hi"}])
|
||||
|
||||
result = asyncio.run(_run())
|
||||
assert result.text == "async hello"
|
||||
assert result.provider == "openai"
|
||||
|
||||
def test_acomplete_structured_parses_json(self):
|
||||
async def fake_async(**_kwargs):
|
||||
return "openai", "gpt-4o", _fake_response('{"x": 42}')
|
||||
|
||||
llm = make_plugin_llm_for_test(
|
||||
plugin_id="my-plugin",
|
||||
policy=_TrustPolicy(plugin_id="my-plugin"),
|
||||
async_caller=fake_async,
|
||||
)
|
||||
|
||||
async def _run() -> PluginLlmStructuredResult:
|
||||
return await llm.acomplete_structured(
|
||||
instructions="Extract x",
|
||||
input=[PluginLlmTextInput(text="data")],
|
||||
json_mode=True,
|
||||
)
|
||||
|
||||
result = asyncio.run(_run())
|
||||
assert result.parsed == {"x": 42}
|
||||
assert result.content_type == "json"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config-driven trust gate (round-trip via plugins.entries.<id>.llm)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestConfigDrivenPolicy:
|
||||
def test_policy_loaded_from_yaml(self, tmp_path, monkeypatch):
|
||||
from agent.plugin_llm import _resolve_trust_policy
|
||||
|
||||
hermes_home = tmp_path / ".hermes"
|
||||
hermes_home.mkdir()
|
||||
(hermes_home / "config.yaml").write_text(
|
||||
"""
|
||||
plugins:
|
||||
entries:
|
||||
my-plugin:
|
||||
llm:
|
||||
allow_provider_override: true
|
||||
allowed_providers: [openrouter, anthropic]
|
||||
allow_model_override: true
|
||||
allowed_models:
|
||||
- openai/gpt-4o-mini
|
||||
- anthropic/claude-3-5-haiku
|
||||
allow_profile_override: false
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
monkeypatch.setenv("HERMES_HOME", str(hermes_home))
|
||||
from hermes_cli import config as _config_mod
|
||||
_config_mod._config_cache = None # type: ignore[attr-defined]
|
||||
|
||||
policy = _resolve_trust_policy("my-plugin")
|
||||
assert policy.allow_provider_override is True
|
||||
assert policy.allow_model_override is True
|
||||
assert policy.allow_profile_override is False
|
||||
assert policy.allowed_providers == frozenset({"openrouter", "anthropic"})
|
||||
assert policy.allowed_models == frozenset({
|
||||
"openai/gpt-4o-mini", "anthropic/claude-3-5-haiku",
|
||||
})
|
||||
|
||||
def test_missing_plugin_entry_yields_default_deny(self, tmp_path, monkeypatch):
|
||||
from agent.plugin_llm import _resolve_trust_policy
|
||||
|
||||
hermes_home = tmp_path / ".hermes"
|
||||
hermes_home.mkdir()
|
||||
(hermes_home / "config.yaml").write_text("plugins: {}\n", encoding="utf-8")
|
||||
monkeypatch.setenv("HERMES_HOME", str(hermes_home))
|
||||
from hermes_cli import config as _config_mod
|
||||
_config_mod._config_cache = None # type: ignore[attr-defined]
|
||||
|
||||
policy = _resolve_trust_policy("never-configured")
|
||||
assert policy.allow_provider_override is False
|
||||
assert policy.allow_model_override is False
|
||||
assert policy.allow_profile_override is False
|
||||
assert policy.allow_agent_id_override is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Plugin context wiring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPluginContextIntegration:
|
||||
def test_ctx_llm_is_lazy_singleton(self):
|
||||
from hermes_cli.plugins import PluginContext, PluginManifest, PluginManager
|
||||
|
||||
manifest = PluginManifest(name="test-plugin", source="test", key="test-plugin")
|
||||
manager = PluginManager()
|
||||
ctx = PluginContext(manifest, manager)
|
||||
first = ctx.llm
|
||||
second = ctx.llm
|
||||
assert first is second
|
||||
assert isinstance(first, PluginLlm)
|
||||
assert first._plugin_id == "test-plugin" # type: ignore[attr-defined]
|
||||
|
||||
def test_ctx_llm_uses_manifest_key_for_policy(self):
|
||||
from hermes_cli.plugins import PluginContext, PluginManifest, PluginManager
|
||||
|
||||
manifest = PluginManifest(
|
||||
name="bare-name", source="test", key="image_gen/openai"
|
||||
)
|
||||
manager = PluginManager()
|
||||
ctx = PluginContext(manifest, manager)
|
||||
assert ctx.llm._plugin_id == "image_gen/openai" # type: ignore[attr-defined]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Attribution (result.provider / result.model / audit log)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAttribution:
|
||||
"""Verifies that the result object and the audit log carry the real
|
||||
provider/model that ``call_llm`` ended up using, NOT the placeholder
|
||||
fallbacks ('auto', 'default') from earlier drafts."""
|
||||
|
||||
def test_explicit_overrides_recorded_when_no_response_model(self):
|
||||
from agent.plugin_llm import _resolve_attribution
|
||||
|
||||
# Response with no .model attribute — overrides win.
|
||||
response = SimpleNamespace(choices=[], usage=None)
|
||||
provider, model = _resolve_attribution(
|
||||
provider_override="openrouter",
|
||||
model_override="anthropic/claude-3-5-sonnet",
|
||||
response=response,
|
||||
)
|
||||
assert provider == "openrouter"
|
||||
assert model == "anthropic/claude-3-5-sonnet"
|
||||
|
||||
def test_response_model_wins_over_model_override(self):
|
||||
"""Providers often canonicalise the model name (e.g. ``gpt-4o``
|
||||
→ ``gpt-4o-2024-08-06``). Whatever they actually returned wins
|
||||
for the recorded model so the audit log reflects reality."""
|
||||
from agent.plugin_llm import _resolve_attribution
|
||||
|
||||
response = SimpleNamespace(model="gpt-4o-2024-08-06", choices=[])
|
||||
provider, model = _resolve_attribution(
|
||||
provider_override="openrouter",
|
||||
model_override="openai/gpt-4o",
|
||||
response=response,
|
||||
)
|
||||
assert model == "gpt-4o-2024-08-06"
|
||||
# Provider override is unaffected by response.model.
|
||||
assert provider == "openrouter"
|
||||
|
||||
def test_falls_back_to_main_provider_and_model_when_no_overrides(self, monkeypatch):
|
||||
"""When the plugin doesn't override anything, attribution
|
||||
reflects the user's active main provider/model rather than
|
||||
misleading placeholders."""
|
||||
from agent import plugin_llm
|
||||
import agent.auxiliary_client as ac
|
||||
|
||||
monkeypatch.setattr(ac, "_read_main_provider", lambda: "openrouter")
|
||||
monkeypatch.setattr(ac, "_read_main_model", lambda: "anthropic/claude-3-5-sonnet")
|
||||
|
||||
response = SimpleNamespace(choices=[]) # no .model attribute
|
||||
provider, model = plugin_llm._resolve_attribution(
|
||||
provider_override=None,
|
||||
model_override=None,
|
||||
response=response,
|
||||
)
|
||||
assert provider == "openrouter"
|
||||
assert model == "anthropic/claude-3-5-sonnet"
|
||||
|
||||
def test_response_model_used_even_when_no_overrides(self, monkeypatch):
|
||||
"""The provider's canonical model name should still flow through
|
||||
when no overrides are set."""
|
||||
from agent import plugin_llm
|
||||
import agent.auxiliary_client as ac
|
||||
|
||||
monkeypatch.setattr(ac, "_read_main_provider", lambda: "openrouter")
|
||||
monkeypatch.setattr(ac, "_read_main_model", lambda: "openai/gpt-4o")
|
||||
|
||||
response = SimpleNamespace(model="openai/gpt-4o-2024-08-06", choices=[])
|
||||
provider, model = plugin_llm._resolve_attribution(
|
||||
provider_override=None,
|
||||
model_override=None,
|
||||
response=response,
|
||||
)
|
||||
assert provider == "openrouter"
|
||||
assert model == "openai/gpt-4o-2024-08-06"
|
||||
|
||||
def test_placeholder_fallback_only_when_everything_is_empty(self, monkeypatch):
|
||||
"""If main_provider/main_model are unset AND there's no override
|
||||
AND the response has no .model, fall through to the safety
|
||||
placeholders so the result object never has empty strings."""
|
||||
from agent import plugin_llm
|
||||
import agent.auxiliary_client as ac
|
||||
|
||||
monkeypatch.setattr(ac, "_read_main_provider", lambda: "")
|
||||
monkeypatch.setattr(ac, "_read_main_model", lambda: "")
|
||||
|
||||
response = SimpleNamespace(choices=[])
|
||||
provider, model = plugin_llm._resolve_attribution(
|
||||
provider_override=None,
|
||||
model_override=None,
|
||||
response=response,
|
||||
)
|
||||
assert provider == "auto"
|
||||
assert model == "default"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hook-mode integration (ctx.llm called from a post_tool_call callback)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestHookMode:
|
||||
"""The docs page promises ``ctx.llm`` works from inside lifecycle
|
||||
hooks. This exercises that path: register a ``post_tool_call``
|
||||
callback that calls ``ctx.llm.complete``, fire the hook through
|
||||
the real ``invoke_hook`` machinery, and check the call landed."""
|
||||
|
||||
def test_complete_works_from_post_tool_call_hook(self):
|
||||
from hermes_cli.plugins import PluginContext, PluginManifest, PluginManager
|
||||
|
||||
manifest = PluginManifest(name="hook-plugin", source="test", key="hook-plugin")
|
||||
manager = PluginManager()
|
||||
ctx = PluginContext(manifest, manager)
|
||||
|
||||
# Replace ctx.llm with a stub that records what the hook called.
|
||||
captured: list = []
|
||||
|
||||
def fake_caller(**kwargs):
|
||||
captured.append(kwargs)
|
||||
return "openrouter", "openai/gpt-4o", _fake_response("rewrote it")
|
||||
|
||||
ctx._llm = make_plugin_llm_for_test( # type: ignore[attr-defined]
|
||||
plugin_id="hook-plugin",
|
||||
policy=_TrustPolicy(plugin_id="hook-plugin"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
|
||||
# Plugin registers a hook that runs ctx.llm.complete on every tool call.
|
||||
def rewrite_error_hook(*, tool_name, args, result, **_):
|
||||
if "Traceback" in (result or ""):
|
||||
rewritten = ctx.llm.complete(
|
||||
messages=[
|
||||
{"role": "system", "content": "Rewrite errors plainly."},
|
||||
{"role": "user", "content": result},
|
||||
],
|
||||
max_tokens=64,
|
||||
purpose="hook-plugin.rewrite",
|
||||
)
|
||||
# Real hook would return the rewritten text via
|
||||
# transform_tool_result; here we just capture for the assert.
|
||||
captured.append({"hook_returned": rewritten.text})
|
||||
|
||||
ctx.register_hook("post_tool_call", rewrite_error_hook)
|
||||
|
||||
# Fire the hook the same way the agent core does it.
|
||||
manager.invoke_hook(
|
||||
"post_tool_call",
|
||||
tool_name="terminal",
|
||||
args={"command": "boom"},
|
||||
result="Traceback (most recent call last):\n RuntimeError",
|
||||
)
|
||||
|
||||
# Verify ctx.llm.complete fired through the hook.
|
||||
assert len(captured) == 2 # one llm call + one hook return record
|
||||
llm_call = captured[0]
|
||||
assert "messages" in llm_call
|
||||
assert any("rewrite" in m.get("content", "").lower()
|
||||
for m in llm_call["messages"] if isinstance(m, dict))
|
||||
hook_record = captured[1]
|
||||
assert hook_record["hook_returned"] == "rewrote it"
|
||||
|
||||
def test_complete_works_from_post_tool_call_hook_when_async_caller_set(self):
|
||||
"""Hooks fired synchronously should still work with sync
|
||||
ctx.llm.complete even if other callsites use async."""
|
||||
from hermes_cli.plugins import PluginContext, PluginManifest, PluginManager
|
||||
|
||||
manifest = PluginManifest(name="hook-async", source="test", key="hook-async")
|
||||
manager = PluginManager()
|
||||
ctx = PluginContext(manifest, manager)
|
||||
|
||||
def fake_caller(**_):
|
||||
return "openrouter", "model-x", _fake_response("ok")
|
||||
|
||||
ctx._llm = make_plugin_llm_for_test( # type: ignore[attr-defined]
|
||||
plugin_id="hook-async",
|
||||
policy=_TrustPolicy(plugin_id="hook-async"),
|
||||
sync_caller=fake_caller,
|
||||
)
|
||||
|
||||
called: list = []
|
||||
|
||||
def hook(**kwargs):
|
||||
r = ctx.llm.complete(messages=[{"role": "user", "content": "x"}])
|
||||
called.append(r.text)
|
||||
|
||||
ctx.register_hook("post_tool_call", hook)
|
||||
manager.invoke_hook("post_tool_call", tool_name="x", args={}, result="y")
|
||||
assert called == ["ok"]
|
||||
465
website/docs/developer-guide/plugin-llm-access.md
Normal file
465
website/docs/developer-guide/plugin-llm-access.md
Normal file
|
|
@ -0,0 +1,465 @@
|
|||
---
|
||||
sidebar_position: 11
|
||||
title: "Plugin LLM Access"
|
||||
description: "Run any LLM call from inside a plugin via ctx.llm — chat or structured, sync or async. Host-owned auth, fail-closed trust gate, optional JSON Schema validation."
|
||||
---
|
||||
|
||||
# Plugin LLM Access
|
||||
|
||||
`ctx.llm` is the supported way for a plugin to make an LLM call.
|
||||
Chat completion, structured extraction, sync, async, with or without
|
||||
images — same surface, same trust gate, same host-owned credentials.
|
||||
|
||||
Plugins reach for this when they need to do something that involves
|
||||
the model but isn't part of the agent's conversation. A hook that
|
||||
rewrites a tool error into something a non-engineer can read. A
|
||||
gateway adapter that translates an inbound message before queuing
|
||||
it. A slash command that summarises a long paste. A scheduled job
|
||||
that scores yesterday's activity and writes one line to a status
|
||||
board. A pre-filter that decides whether a message is worth waking
|
||||
the agent up for at all.
|
||||
|
||||
These are jobs the agent shouldn't be in the loop on. They want one
|
||||
LLM call, a typed answer, and to be done.
|
||||
|
||||
## The smallest possible call
|
||||
|
||||
```python
|
||||
result = ctx.llm.complete(messages=[{"role": "user", "content": "ping"}])
|
||||
return result.text
|
||||
```
|
||||
|
||||
That's the whole API in one line. No keys, no provider config, no
|
||||
SDK initialisation. The plugin runs against whatever provider and
|
||||
model the user is currently using — when they switch providers, the
|
||||
plugin follows them automatically.
|
||||
|
||||
## A more complete chat example
|
||||
|
||||
```python
|
||||
result = ctx.llm.complete(
|
||||
messages=[
|
||||
{"role": "system", "content": "Rewrite errors as one short sentence a non-engineer can act on."},
|
||||
{"role": "user", "content": traceback_text},
|
||||
],
|
||||
max_tokens=64,
|
||||
purpose="hooks.error-rewrite",
|
||||
)
|
||||
return result.text
|
||||
```
|
||||
|
||||
`purpose` is a free-form audit string — it shows up in `agent.log`
|
||||
and in `result.audit` so operators can see which plugin made which
|
||||
call. Optional but recommended for anything that fires often.
|
||||
|
||||
## Structured output
|
||||
|
||||
When the plugin needs a typed answer, switch to the structured lane:
|
||||
|
||||
```python
|
||||
result = ctx.llm.complete_structured(
|
||||
instructions="Score this support reply for urgency (0–1) and pick a category.",
|
||||
input=[{"type": "text", "text": message_body}],
|
||||
json_schema=TRIAGE_SCHEMA,
|
||||
purpose="support.triage",
|
||||
temperature=0.0,
|
||||
max_tokens=128,
|
||||
)
|
||||
|
||||
if result.parsed["urgency"] > 0.8:
|
||||
await dispatch_to_oncall(result.parsed["category"], message_body)
|
||||
```
|
||||
|
||||
The host requests JSON output from the provider, parses it locally
|
||||
as a fallback, validates against your schema if `jsonschema` is
|
||||
installed, and hands back a Python object on `result.parsed`. If the
|
||||
model couldn't produce valid JSON, `result.parsed` is `None` and
|
||||
`result.text` carries the raw response.
|
||||
|
||||
## What this lane gives you
|
||||
|
||||
* **One call, four shapes.** `complete()` for chat,
|
||||
`complete_structured()` for typed JSON, `acomplete()` and
|
||||
`acomplete_structured()` for asyncio. Same arguments, same result
|
||||
objects.
|
||||
* **Host-owned credentials.** OAuth tokens, refresh flows, the
|
||||
credential pool, per-task aux overrides — every credential
|
||||
concept Hermes already has applies. The plugin never sees a
|
||||
token; the host attributes the call back through `result.audit`.
|
||||
* **Bounded.** Single sync or async call. No streaming, no tool
|
||||
loops, no conversation state to manage. State the input, get the
|
||||
result, return.
|
||||
* **Fail-closed trust.** A plugin you've never configured cannot
|
||||
pick its own provider, model, agent, or stored credential. The
|
||||
default posture is "use what the user is using." Operators opt in
|
||||
to specific overrides, per plugin, in `config.yaml`.
|
||||
|
||||
## Quick start
|
||||
|
||||
Two complete plugins below — one chat, one structured. Both ship
|
||||
inside a single `register(ctx)` function and need zero outside
|
||||
configuration to run against whatever model the user has active.
|
||||
|
||||
### Chat completion — `/tldr`
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_command(
|
||||
name="tldr",
|
||||
handler=lambda raw: _tldr(ctx, raw),
|
||||
description="Summarise the supplied text in one paragraph.",
|
||||
args_hint="<text>",
|
||||
)
|
||||
|
||||
|
||||
def _tldr(ctx, raw_args: str) -> str:
|
||||
text = raw_args.strip()
|
||||
if not text:
|
||||
return "Usage: /tldr <text to summarise>"
|
||||
result = ctx.llm.complete(
|
||||
messages=[
|
||||
{"role": "system",
|
||||
"content": "Summarise the user's text in one tight paragraph. No preamble."},
|
||||
{"role": "user", "content": text},
|
||||
],
|
||||
max_tokens=256,
|
||||
temperature=0.3,
|
||||
purpose="tldr",
|
||||
)
|
||||
return result.text
|
||||
```
|
||||
|
||||
`result.text` is the model's response; `result.usage` carries token
|
||||
counts; `result.provider` and `result.model` carry attribution.
|
||||
|
||||
### Structured extraction — `/paste-to-tasks`
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_command(
|
||||
name="paste-to-tasks",
|
||||
handler=lambda raw: _paste_to_tasks(ctx, raw),
|
||||
description="Turn freeform meeting notes into structured tasks.",
|
||||
args_hint="<text>",
|
||||
)
|
||||
|
||||
|
||||
_TASKS_SCHEMA = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"tasks": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"owner": {"type": "string"},
|
||||
"action": {"type": "string"},
|
||||
"due": {"type": "string", "description": "ISO date or empty"},
|
||||
},
|
||||
"required": ["action"],
|
||||
},
|
||||
},
|
||||
},
|
||||
"required": ["tasks"],
|
||||
}
|
||||
|
||||
|
||||
def _paste_to_tasks(ctx, raw_args: str) -> str:
|
||||
if not raw_args.strip():
|
||||
return "Usage: /paste-to-tasks <meeting notes>"
|
||||
result = ctx.llm.complete_structured(
|
||||
instructions=(
|
||||
"Extract concrete action items from these meeting notes. "
|
||||
"One task per actionable line. If no owner is named, leave 'owner' blank."
|
||||
),
|
||||
input=[{"type": "text", "text": raw_args}],
|
||||
json_schema=_TASKS_SCHEMA,
|
||||
schema_name="meeting.tasks",
|
||||
purpose="paste-to-tasks",
|
||||
temperature=0.0,
|
||||
max_tokens=512,
|
||||
)
|
||||
if result.parsed is None:
|
||||
return f"Couldn't parse a response. Raw output:\n{result.text}"
|
||||
lines = [f"- [{t.get('owner') or '?'}] {t['action']}" for t in result.parsed["tasks"]]
|
||||
return "\n".join(lines) or "(no tasks found)"
|
||||
```
|
||||
|
||||
A third worked example, this time with image input, lives in the
|
||||
[`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-example)
|
||||
repo (companion repo for reference plugins — not bundled with
|
||||
hermes-agent itself). For the async surface (`acomplete()` /
|
||||
`acomplete_structured()` with `asyncio.gather()`), see
|
||||
[`plugin-llm-async-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-async-example)
|
||||
in the same repo.
|
||||
|
||||
## When to use which
|
||||
|
||||
| You want… | Reach for |
|
||||
|---|---|
|
||||
| A free-form text response (translation, summary, rewrite, generation) | `complete()` |
|
||||
| A multi-turn prompt (system + few-shot examples + user) | `complete()` |
|
||||
| A typed dict back, validated against a schema | `complete_structured()` |
|
||||
| Image-or-text input with a typed dict back | `complete_structured()` |
|
||||
| The same call from async code (gateway adapters, async hooks) | `acomplete()` / `acomplete_structured()` |
|
||||
|
||||
Everything else — provider selection, model resolution, auth, fallback,
|
||||
timeout, vision routing — is the same across all four.
|
||||
|
||||
## API surface
|
||||
|
||||
`ctx.llm` is an instance of `agent.plugin_llm.PluginLlm`.
|
||||
|
||||
### `complete()`
|
||||
|
||||
```python
|
||||
result = ctx.llm.complete(
|
||||
messages=[{"role": "user", "content": "Hi"}],
|
||||
provider=None, # optional, gated — Hermes provider id (e.g. "openrouter")
|
||||
model=None, # optional, gated — whatever string that provider expects
|
||||
temperature=None,
|
||||
max_tokens=None,
|
||||
timeout=None, # seconds
|
||||
agent_id=None, # optional, gated
|
||||
profile=None, # optional, gated — explicit auth-profile name
|
||||
purpose="optional-audit-string",
|
||||
)
|
||||
# → PluginLlmCompleteResult(text, provider, model, agent_id, usage, audit)
|
||||
```
|
||||
|
||||
Plain chat completion. `messages` is the standard OpenAI shape — a
|
||||
list of `{"role": "...", "content": "..."}` dicts. Multi-turn
|
||||
prompts (system + few-shot user/assistant pairs + final user) work
|
||||
exactly as they would with the OpenAI SDK.
|
||||
|
||||
`provider=` and `model=` are independent and follow the same shape
|
||||
as the host's main config (`model.provider` + `model.model`). Set
|
||||
just `model=` to use the user's active provider with a different
|
||||
model on it. Set both to switch providers entirely. Either argument
|
||||
without operator opt-in raises `PluginLlmTrustError`.
|
||||
|
||||
### `complete_structured()`
|
||||
|
||||
```python
|
||||
result = ctx.llm.complete_structured(
|
||||
instructions="What you want extracted.",
|
||||
input=[
|
||||
{"type": "text", "text": "..."},
|
||||
{"type": "image", "data": b"...", "mime_type": "image/png"},
|
||||
{"type": "image", "url": "https://..."},
|
||||
],
|
||||
json_schema={...}, # optional — triggers parsed result + validation
|
||||
json_mode=False, # set True without a schema to ask for JSON anyway
|
||||
schema_name=None, # optional human-readable schema name
|
||||
system_prompt=None,
|
||||
provider=None, # optional, gated
|
||||
model=None, # optional, gated
|
||||
temperature=None,
|
||||
max_tokens=None,
|
||||
timeout=None,
|
||||
agent_id=None,
|
||||
profile=None,
|
||||
purpose=None,
|
||||
)
|
||||
# → PluginLlmStructuredResult(text, provider, model, agent_id,
|
||||
# usage, parsed, content_type, audit)
|
||||
```
|
||||
|
||||
Inputs are typed text or image blocks (raw bytes get base64 encoded
|
||||
as a `data:` URL automatically). When `json_schema` or
|
||||
`json_mode=True` is supplied, the host requests JSON output via
|
||||
`response_format`, parses it locally as a fallback, and validates
|
||||
against your schema if `jsonschema` is installed.
|
||||
|
||||
* `result.content_type == "json"` — `result.parsed` is a Python
|
||||
object that matches your schema.
|
||||
* `result.content_type == "text"` — parsing or validation failed;
|
||||
inspect `result.text` for the raw model response.
|
||||
|
||||
### Async
|
||||
|
||||
```python
|
||||
result = await ctx.llm.acomplete(messages=...)
|
||||
result = await ctx.llm.acomplete_structured(instructions=..., input=...)
|
||||
```
|
||||
|
||||
Same arguments and result types as their sync counterparts. Use
|
||||
these from gateway adapters, async hooks, or any plugin code
|
||||
already running on an asyncio loop.
|
||||
|
||||
### Result attributes
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class PluginLlmCompleteResult:
|
||||
text: str # the assistant's response
|
||||
provider: str # e.g. "openrouter", "anthropic"
|
||||
model: str # whatever the provider returned for this call
|
||||
agent_id: str # whose model/auth was used
|
||||
usage: PluginLlmUsage # tokens + cache + cost estimate
|
||||
audit: Dict[str, Any] # plugin_id, purpose, profile
|
||||
|
||||
@dataclass
|
||||
class PluginLlmStructuredResult(PluginLlmCompleteResult):
|
||||
parsed: Optional[Any] # JSON object when content_type == "json"
|
||||
content_type: str # "json" or "text"
|
||||
# audit also carries schema_name when supplied
|
||||
```
|
||||
|
||||
`usage` carries `input_tokens`, `output_tokens`, `total_tokens`,
|
||||
`cache_read_tokens`, `cache_write_tokens`, and `cost_usd` when the
|
||||
provider returns those fields.
|
||||
|
||||
## Trust gate
|
||||
|
||||
The default behaviour is fail-closed. With no `plugins.entries`
|
||||
config block, a plugin can:
|
||||
|
||||
* run any of the four methods against the user's active provider
|
||||
and model,
|
||||
* set request-shaping arguments (`temperature`, `max_tokens`,
|
||||
`timeout`, `system_prompt`, `purpose`, `messages`, `instructions`,
|
||||
`input`, `json_schema`),
|
||||
|
||||
…and that's it. `provider=`, `model=`, `agent_id=`, and `profile=`
|
||||
arguments raise `PluginLlmTrustError` until the operator opts in.
|
||||
|
||||
**Most plugins never need this section.** A plugin that just calls
|
||||
`ctx.llm.complete(messages=...)` with no overrides runs against
|
||||
whatever the user has active and works zero-config. The block below
|
||||
is only relevant when a plugin specifically wants to pin to a
|
||||
different model or provider than the user.
|
||||
|
||||
```yaml
|
||||
plugins:
|
||||
entries:
|
||||
my-plugin:
|
||||
llm:
|
||||
# Allow this plugin to choose a different Hermes provider
|
||||
# (must be one Hermes already knows about — same names as
|
||||
# `hermes model` and config.yaml model.provider).
|
||||
allow_provider_override: true
|
||||
|
||||
# Optionally restrict which providers. Use ["*"] for any.
|
||||
allowed_providers:
|
||||
- openrouter
|
||||
- anthropic
|
||||
|
||||
# Allow this plugin to ask for a specific model.
|
||||
allow_model_override: true
|
||||
|
||||
# Optionally restrict which models. Use ["*"] for any.
|
||||
# Models are matched literally against whatever string the
|
||||
# plugin sends — Hermes does not look anything up.
|
||||
allowed_models:
|
||||
- openai/gpt-4o-mini
|
||||
- anthropic/claude-3-5-haiku
|
||||
|
||||
# Allow cross-agent calls (rare).
|
||||
allow_agent_id_override: false
|
||||
|
||||
# Allow the plugin to request a specific stored auth profile
|
||||
# (e.g. a different OAuth account on the same provider).
|
||||
allow_profile_override: false
|
||||
```
|
||||
|
||||
The plugin id is the manifest `name:` field for flat plugins, or the
|
||||
path-derived key for nested plugins (`image_gen/openai`,
|
||||
`memory/honcho`, etc.).
|
||||
|
||||
### What the gate enforces
|
||||
|
||||
| Override | Default | Config key |
|
||||
| --------------- | ------- | -------------------------------- |
|
||||
| `provider=` | denied | `allow_provider_override: true` |
|
||||
| ↳ allowlist | — | `allowed_providers: [...]` |
|
||||
| `model=` | denied | `allow_model_override: true` |
|
||||
| ↳ allowlist | — | `allowed_models: [...]` |
|
||||
| `agent_id=` | denied | `allow_agent_id_override: true` |
|
||||
| `profile=` | denied | `allow_profile_override: true` |
|
||||
|
||||
Each override is independently gated. Granting `allow_model_override`
|
||||
does **not** also grant `allow_provider_override` — a plugin trusted
|
||||
to pick a model is still pinned to the user's active provider unless
|
||||
it gets the provider gate as well.
|
||||
|
||||
### What the gate does NOT need to enforce
|
||||
|
||||
* Request-shaping arguments — `temperature`, `max_tokens`,
|
||||
`timeout`, `system_prompt`, `purpose`, `messages`, `instructions`,
|
||||
`input`, `json_schema`, `schema_name`, `json_mode` — are always
|
||||
allowed; they don't pick credentials or routes.
|
||||
* The default deny posture means an unconfigured plugin can still do
|
||||
useful work — it just runs against the active provider and model.
|
||||
Operators only need to think about `plugins.entries` for plugins
|
||||
that want finer routing.
|
||||
|
||||
## What the host owns
|
||||
|
||||
A complete list of the things `ctx.llm` does for the plugin so you
|
||||
don't have to:
|
||||
|
||||
* **Provider resolution.** Reads `model.provider` + `model.model`
|
||||
from the user's config (or the explicit overrides when trusted).
|
||||
* **Auth.** Pulls API keys, OAuth tokens, or refresh tokens from
|
||||
`~/.hermes/auth.json` / env, including the credential pool when
|
||||
one is configured. The plugin never sees them.
|
||||
* **Vision routing.** When image input is supplied and the user's
|
||||
active text model is text-only, the host falls back to the
|
||||
configured vision model automatically.
|
||||
* **Fallback chain.** If the user's primary provider 5xxs or 429s,
|
||||
the request goes through Hermes' usual aggregator-aware fallback
|
||||
before it returns an error to the plugin.
|
||||
* **Timeout.** Honours your `timeout=` argument, falling back to
|
||||
`auxiliary.<task>.timeout` config or the global aux default.
|
||||
* **JSON shaping.** Sends `response_format` to the provider when
|
||||
you ask for JSON, then re-parses locally from a code-fenced
|
||||
response if the provider returned one.
|
||||
* **Schema validation.** Validates against your `json_schema` when
|
||||
`jsonschema` is installed; logs a debug line and skips strict
|
||||
validation otherwise.
|
||||
* **Audit log.** Each call writes one INFO line to `agent.log` with
|
||||
the plugin id, provider/model, purpose, and token totals.
|
||||
|
||||
## What the plugin owns
|
||||
|
||||
* **Request shape.** `messages` for chat, `instructions` + `input`
|
||||
for structured. The plugin builds the prompt; the host runs it.
|
||||
* **Schema.** Whatever shape you want back. The host doesn't infer
|
||||
it for you.
|
||||
* **Error handling.** `complete_structured()` raises `ValueError` on
|
||||
empty inputs and on schema-validation failure. `PluginLlmTrustError`
|
||||
fires when the trust gate denies an override. Anything else
|
||||
(provider 5xx, no credentials configured, timeout) raises whatever
|
||||
`auxiliary_client.call_llm()` raises.
|
||||
* **Cost.** Every call runs against the user's paid provider. Don't
|
||||
loop on `complete()` for every gateway message without thinking
|
||||
about token spend.
|
||||
|
||||
## Where this fits in the plugin surface
|
||||
|
||||
Existing `ctx.*` methods extend an existing Hermes subsystem:
|
||||
|
||||
| `ctx.register_tool` | adds a tool the agent can call |
|
||||
| `ctx.register_platform` | wires a new gateway adapter |
|
||||
| `ctx.register_image_gen_provider` | replaces an image-gen backend |
|
||||
| `ctx.register_memory_provider` | replaces the memory backend |
|
||||
| `ctx.register_context_engine` | replaces the context compressor |
|
||||
| `ctx.register_hook` | observes a lifecycle event |
|
||||
|
||||
`ctx.llm` is the first surface that lets a plugin run the same
|
||||
model the user is talking to, *out of band*, without any of the
|
||||
above. That's its only job. If your plugin needs to register a
|
||||
tool the agent invokes, use `register_tool`. If it needs to react
|
||||
to a lifecycle event, use `register_hook`. If it needs to make its
|
||||
own model call — for any reason, structured or not — `ctx.llm`.
|
||||
|
||||
## Reference
|
||||
|
||||
* Implementation: [`agent/plugin_llm.py`](https://github.com/NousResearch/hermes-agent/blob/main/agent/plugin_llm.py)
|
||||
* Tests: [`tests/agent/test_plugin_llm.py`](https://github.com/NousResearch/hermes-agent/blob/main/tests/agent/test_plugin_llm.py)
|
||||
* Reference plugins (companion repo):
|
||||
* [`plugin-llm-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-example) — sync structured extraction with image input
|
||||
* [`plugin-llm-async-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-async-example) — async with `asyncio.gather()`
|
||||
* Auxiliary client (the engine under the hood): see
|
||||
[Provider Runtime](/docs/developer-guide/provider-runtime).
|
||||
|
|
@ -64,8 +64,6 @@ The repo ships these bundled plugins under `plugins/`. All are opt-in — enable
|
|||
| `image_gen/xai` | image backend | xAI `grok-2-image` backend |
|
||||
| `hermes-achievements` | dashboard tab | Steam-style collectible badges generated from your real Hermes session history |
|
||||
| `kanban/dashboard` | dashboard tab | Kanban board UI for the multi-agent dispatcher — tasks, comments, fan-out, board switching. See [Kanban Multi-Agent](./kanban.md). |
|
||||
| `example-dashboard` | dashboard example | Reference dashboard plugin for [Extending the Dashboard](./extending-the-dashboard.md) |
|
||||
| `strike-freedom-cockpit` | dashboard skin | Sample custom dashboard skin |
|
||||
|
||||
Memory providers (`plugins/memory/*`) and context engines (`plugins/context_engine/*`) are listed separately on [Memory Providers](./memory-providers.md) — they're managed through `hermes memory` and `hermes plugins` respectively. The full per-plugin detail for the two long-running hooks-based plugins follows.
|
||||
|
||||
|
|
|
|||
|
|
@ -681,7 +681,7 @@ Key points:
|
|||
- Multiple plugins can claim the same page-scoped slot. They render stacked in registration order.
|
||||
- Zero footprint when no plugin registers: the built-in page renders exactly as before.
|
||||
|
||||
The bundled `example-dashboard` plugin ships a live demo that injects a banner into `sessions:top` — install it to see the pattern end-to-end.
|
||||
A reference plugin (`example-dashboard` in [`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins/tree/main/example-dashboard)) ships a live demo that injects a banner into `sessions:top` — install it to see the pattern end-to-end.
|
||||
|
||||
### Slot-only plugins (`tab.hidden`)
|
||||
|
||||
|
|
@ -818,7 +818,7 @@ If a plugin's script fails to load (404, syntax error, exception during IIFE), t
|
|||
|
||||
## Combined theme + plugin demo
|
||||
|
||||
The repo ships `plugins/strike-freedom-cockpit/` as a complete reskin demo. It pairs a theme YAML with a slot-only plugin to produce a cockpit-style HUD without forking the dashboard.
|
||||
The [`strike-freedom-cockpit`](https://github.com/NousResearch/hermes-example-plugins/tree/main/strike-freedom-cockpit) plugin (companion repo `hermes-example-plugins`) is a complete reskin demo. It pairs a theme YAML with a slot-only plugin to produce a cockpit-style HUD without forking the dashboard.
|
||||
|
||||
**What it demonstrates:**
|
||||
|
||||
|
|
@ -832,17 +832,19 @@ The repo ships `plugins/strike-freedom-cockpit/` as a complete reskin demo. It p
|
|||
**Install:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/NousResearch/hermes-example-plugins.git
|
||||
|
||||
# Theme
|
||||
cp plugins/strike-freedom-cockpit/theme/strike-freedom.yaml \
|
||||
cp hermes-example-plugins/strike-freedom-cockpit/theme/strike-freedom.yaml \
|
||||
~/.hermes/dashboard-themes/
|
||||
|
||||
# Plugin
|
||||
cp -r plugins/strike-freedom-cockpit ~/.hermes/plugins/
|
||||
cp -r hermes-example-plugins/strike-freedom-cockpit ~/.hermes/plugins/
|
||||
```
|
||||
|
||||
Open the dashboard, pick **Strike Freedom** from the theme switcher. The cockpit sidebar appears, the crest shows in the header, the tagline replaces the footer. Switch back to **Hermes Teal** and the plugin remains installed but invisible (the `sidebar` slot only renders under the `cockpit` layout variant).
|
||||
|
||||
Read the plugin source (`plugins/strike-freedom-cockpit/dashboard/dist/index.js`) to see how it reads CSS vars, guards against older dashboards without slot support, and registers three slots from one bundle.
|
||||
Read the plugin source (`strike-freedom-cockpit/dashboard/dist/index.js` in the companion repo) to see how it reads CSS vars, guards against older dashboards without slot support, and registers three slots from one bundle.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -111,6 +111,7 @@ Every `ctx.*` API below is available inside a plugin's `register(ctx)` function.
|
|||
| Register an image-generation backend | `ctx.register_image_gen_provider(provider)` — see [Image Generation Provider Plugins](/docs/developer-guide/image-gen-provider-plugin) |
|
||||
| Register a context-compression engine | `ctx.register_context_engine(engine)` — see [Context Engine Plugins](/docs/developer-guide/context-engine-plugin) |
|
||||
| Register a memory backend | Subclass `MemoryProvider` in `plugins/memory/<name>/__init__.py` — see [Memory Provider Plugins](/docs/developer-guide/memory-provider-plugin) (uses a separate discovery system) |
|
||||
| Run a host-owned LLM call | `ctx.llm.complete(...)` / `ctx.llm.complete_structured(...)` — borrow the user's active model + auth for a one-shot completion with optional JSON schema validation. See [Plugin LLM Access](/docs/developer-guide/plugin-llm-access) |
|
||||
| Register an inference backend (LLM provider) | `register_provider(ProviderProfile(...))` in `plugins/model-providers/<name>/__init__.py` — see [Model Provider Plugins](/docs/developer-guide/model-provider-plugin) (uses a separate discovery system) |
|
||||
|
||||
## Plugin discovery
|
||||
|
|
|
|||
|
|
@ -221,6 +221,7 @@ const sidebars: SidebarsConfig = {
|
|||
'developer-guide/context-engine-plugin',
|
||||
'developer-guide/model-provider-plugin',
|
||||
'developer-guide/image-gen-provider-plugin',
|
||||
'developer-guide/plugin-llm-access',
|
||||
'developer-guide/creating-skills',
|
||||
'developer-guide/extending-the-cli',
|
||||
],
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue