mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-25 00:51:20 +00:00
* feat(plugins): pluggable image_gen backends + OpenAI provider
Adds a ImageGenProvider ABC so image generation backends register as
bundled plugins under `plugins/image_gen/<name>/`. The plugin scanner
gains three primitives to make this work generically:
- `kind:` manifest field (`standalone` | `backend` | `exclusive`).
Bundled `kind: backend` plugins auto-load — no `plugins.enabled`
incantation. User-installed backends stay opt-in.
- Path-derived keys: `plugins/image_gen/openai/` gets key
`image_gen/openai`, so a future `tts/openai` cannot collide.
- Depth-2 recursion into category namespaces (parent dirs without a
`plugin.yaml` of their own).
Includes `OpenAIImageGenProvider` as the first consumer (gpt-image-1.5
default, plus gpt-image-1, gpt-image-1-mini, DALL-E 3/2). Base64
responses save to `$HERMES_HOME/cache/images/`; URL responses pass
through.
FAL stays in-tree for this PR — a follow-up ports it into
`plugins/image_gen/fal/` so the in-tree `image_generation_tool.py`
slims down. The dispatch shim in `_handle_image_generate` only fires
when `image_gen.provider` is explicitly set to a non-FAL value, so
existing FAL setups are untouched.
- 41 unit tests (scanner recursion, kind parsing, gate logic,
registry, OpenAI payload shapes)
- E2E smoke verified: bundled plugin autoloads, registers, and
`_handle_image_generate` routes to OpenAI when configured
* fix(image_gen/openai): don't send response_format to gpt-image-*
The live API rejects it: 'Unknown parameter: response_format'
(verified 2026-04-21 with gpt-image-1.5). gpt-image-* models return
b64_json unconditionally, so the parameter was both unnecessary and
actively broken.
* feat(image_gen/openai): gpt-image-2 only, drop legacy catalog
gpt-image-2 is the latest/best OpenAI image model (released 2026-04-21)
and there's no reason to expose the older gpt-image-1.5 / gpt-image-1 /
dall-e-3 / dall-e-2 alongside it — slower, lower quality, or awkward
(dall-e-2 squares only). Trim the catalog down to a single model.
Live-verified end-to-end: landscape 1536x1024 render of a Moog-style
synth matches prompt exactly, 2.4MB PNG saved to cache.
* feat(image_gen/openai): expose gpt-image-2 as three quality tiers
Users pick speed/fidelity via the normal model picker instead of a
hidden quality knob. All three tier IDs resolve to the single underlying
gpt-image-2 API model with a different quality parameter:
gpt-image-2-low ~15s fast iteration
gpt-image-2-medium ~40s default
gpt-image-2-high ~2min highest fidelity
Live-measured on OpenAI's API today: 15.4s / 40.8s / 116.9s for the
same 1024x1024 prompt.
Config:
image_gen.openai.model: gpt-image-2-high
# or
image_gen.model: gpt-image-2-low
# or env var for scripts/tests
OPENAI_IMAGE_MODEL=gpt-image-2-medium
Live-verified end-to-end with the low tier: 18.8s landscape render of a
golden retriever in wildflowers, vision-confirmed exact match.
* feat(tools_config): plugin image_gen providers inject themselves into picker
'hermes tools' → Image Generation now shows plugin-registered backends
alongside Nous Subscription and FAL.ai without tools_config.py needing
to know about them. OpenAI appears as a third option today; future
backends appear automatically as they're added.
Mechanism:
- ImageGenProvider gains an optional get_setup_schema() hook
(name, badge, tag, env_vars). Default derived from display_name.
- tools_config._plugin_image_gen_providers() pulls the schemas from
every registered non-FAL plugin provider.
- _visible_providers() appends those rows when rendering the Image
Generation category.
- _configure_provider() handles the new image_gen_plugin_name marker:
writes image_gen.provider and routes to the plugin's list_models()
catalog for the model picker.
- _toolset_needs_configuration_prompt('image_gen') stops demanding a
FAL key when any plugin provider reports is_available().
FAL is skipped in the plugin path because it already has hardcoded
TOOL_CATEGORIES rows — when it gets ported to a plugin in a follow-up
PR the hardcoded rows go away and it surfaces through the same path
as OpenAI.
Verified live: picker shows Nous Subscription / FAL.ai / OpenAI.
Picking OpenAI prompts for OPENAI_API_KEY, then shows the
gpt-image-2-low/medium/high model picker sourced from the plugin.
397 tests pass across plugins/, tools_config, registry, and picker.
* fix(image_gen): close final gaps for plugin-backend parity with FAL
Two small places that still hardcoded FAL:
- hermes_cli/setup.py status line: an OpenAI-only setup showed
'Image Generation: missing FAL_KEY'. Now probes plugin providers
and reports '(OpenAI)' when one is_available() — or falls back to
'missing FAL_KEY or OPENAI_API_KEY' if nothing is configured.
- image_generate tool schema description: said 'using FAL.ai, default
FLUX 2 Klein 9B'. Rewrote provider-neutral — 'backend and model are
user-configured' — and notes the 'image' field can be a URL or an
absolute path, which the gateway delivers either way via
extract_local_files().
174 lines
6.2 KiB
Python
174 lines
6.2 KiB
Python
"""Tests for plugin image_gen providers injecting themselves into the picker.
|
|
|
|
Covers `_plugin_image_gen_providers`, `_visible_providers`, and
|
|
`_toolset_needs_configuration_prompt` handling of plugin providers.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from agent import image_gen_registry
|
|
from agent.image_gen_provider import ImageGenProvider
|
|
|
|
|
|
class _FakeProvider(ImageGenProvider):
|
|
def __init__(self, name: str, available: bool = True, schema=None, models=None):
|
|
self._name = name
|
|
self._available = available
|
|
self._schema = schema or {
|
|
"name": name.title(),
|
|
"badge": "test",
|
|
"tag": f"{name} test tag",
|
|
"env_vars": [{"key": f"{name.upper()}_API_KEY", "prompt": f"{name} key"}],
|
|
}
|
|
self._models = models or [
|
|
{"id": f"{name}-model-v1", "display": f"{name} v1",
|
|
"speed": "~5s", "strengths": "test", "price": "$"},
|
|
]
|
|
|
|
@property
|
|
def name(self) -> str:
|
|
return self._name
|
|
|
|
def is_available(self) -> bool:
|
|
return self._available
|
|
|
|
def list_models(self):
|
|
return list(self._models)
|
|
|
|
def default_model(self):
|
|
return self._models[0]["id"] if self._models else None
|
|
|
|
def get_setup_schema(self):
|
|
return dict(self._schema)
|
|
|
|
def generate(self, prompt, aspect_ratio="landscape", **kw):
|
|
return {"success": True, "image": f"{self._name}://{prompt}"}
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _reset_registry():
|
|
image_gen_registry._reset_for_tests()
|
|
yield
|
|
image_gen_registry._reset_for_tests()
|
|
|
|
|
|
class TestPluginPickerInjection:
|
|
def test_plugin_providers_returns_registered(self, monkeypatch):
|
|
from hermes_cli import tools_config
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("myimg"))
|
|
|
|
rows = tools_config._plugin_image_gen_providers()
|
|
names = [r["name"] for r in rows]
|
|
plugin_names = [r.get("image_gen_plugin_name") for r in rows]
|
|
|
|
assert "Myimg" in names
|
|
assert "myimg" in plugin_names
|
|
|
|
def test_fal_skipped_to_avoid_duplicate(self, monkeypatch):
|
|
from hermes_cli import tools_config
|
|
|
|
# Simulate a FAL plugin being registered — the picker already has
|
|
# hardcoded FAL rows in TOOL_CATEGORIES, so plugin-FAL must be
|
|
# skipped to avoid showing FAL twice.
|
|
image_gen_registry.register_provider(_FakeProvider("fal"))
|
|
image_gen_registry.register_provider(_FakeProvider("openai"))
|
|
|
|
rows = tools_config._plugin_image_gen_providers()
|
|
names = [r.get("image_gen_plugin_name") for r in rows]
|
|
assert "fal" not in names
|
|
assert "openai" in names
|
|
|
|
def test_visible_providers_includes_plugins_for_image_gen(self, monkeypatch):
|
|
from hermes_cli import tools_config
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("someimg"))
|
|
|
|
cat = tools_config.TOOL_CATEGORIES["image_gen"]
|
|
visible = tools_config._visible_providers(cat, {})
|
|
plugin_names = [p.get("image_gen_plugin_name") for p in visible if p.get("image_gen_plugin_name")]
|
|
assert "someimg" in plugin_names
|
|
|
|
def test_visible_providers_does_not_inject_into_other_categories(self, monkeypatch):
|
|
from hermes_cli import tools_config
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("someimg"))
|
|
|
|
# Browser category must NOT see image_gen plugins.
|
|
browser = tools_config.TOOL_CATEGORIES["browser"]
|
|
visible = tools_config._visible_providers(browser, {})
|
|
assert all(p.get("image_gen_plugin_name") is None for p in visible)
|
|
|
|
|
|
class TestPluginCatalog:
|
|
def test_plugin_catalog_returns_models(self):
|
|
from hermes_cli import tools_config
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("catimg"))
|
|
|
|
catalog, default = tools_config._plugin_image_gen_catalog("catimg")
|
|
assert "catimg-model-v1" in catalog
|
|
assert default == "catimg-model-v1"
|
|
|
|
def test_plugin_catalog_empty_for_unknown(self):
|
|
from hermes_cli import tools_config
|
|
|
|
catalog, default = tools_config._plugin_image_gen_catalog("does-not-exist")
|
|
assert catalog == {}
|
|
assert default is None
|
|
|
|
|
|
class TestConfigPrompt:
|
|
def test_image_gen_satisfied_by_plugin_provider(self, monkeypatch, tmp_path):
|
|
"""When a plugin provider reports is_available(), the picker should
|
|
not force a setup prompt on the user."""
|
|
from hermes_cli import tools_config
|
|
|
|
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
|
|
monkeypatch.delenv("FAL_KEY", raising=False)
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("avail-img", available=True))
|
|
|
|
assert tools_config._toolset_needs_configuration_prompt("image_gen", {}) is False
|
|
|
|
def test_image_gen_still_prompts_when_nothing_available(self, monkeypatch, tmp_path):
|
|
from hermes_cli import tools_config
|
|
|
|
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
|
|
monkeypatch.delenv("FAL_KEY", raising=False)
|
|
|
|
image_gen_registry.register_provider(_FakeProvider("unavail-img", available=False))
|
|
|
|
assert tools_config._toolset_needs_configuration_prompt("image_gen", {}) is True
|
|
|
|
|
|
class TestConfigWriting:
|
|
def test_picking_plugin_provider_writes_provider_and_model(self, monkeypatch, tmp_path):
|
|
"""When a user picks a plugin-backed image_gen provider with no
|
|
env vars needed, ``_configure_provider`` should write both
|
|
``image_gen.provider`` and ``image_gen.model``."""
|
|
from hermes_cli import tools_config
|
|
|
|
monkeypatch.setenv("HERMES_HOME", str(tmp_path))
|
|
image_gen_registry.register_provider(_FakeProvider("noenv", schema={
|
|
"name": "NoEnv",
|
|
"badge": "free",
|
|
"tag": "",
|
|
"env_vars": [],
|
|
}))
|
|
|
|
# Stub out the interactive model picker — no TTY in tests.
|
|
monkeypatch.setattr(tools_config, "_prompt_choice", lambda *a, **kw: 0)
|
|
|
|
config: dict = {}
|
|
provider_row = {
|
|
"name": "NoEnv",
|
|
"env_vars": [],
|
|
"image_gen_plugin_name": "noenv",
|
|
}
|
|
tools_config._configure_provider(provider_row, config)
|
|
|
|
assert config["image_gen"]["provider"] == "noenv"
|
|
assert config["image_gen"]["model"] == "noenv-model-v1"
|