mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-25 00:51:20 +00:00
feat(image_gen): multi-model FAL support with picker in hermes tools (#11265)
* feat(image_gen): multi-model FAL support with picker in hermes tools
Adds 8 FAL text-to-image models selectable via `hermes tools` →
Image Generation → (FAL.ai | Nous Subscription) → model picker.
Models supported:
- fal-ai/flux-2/klein/9b (new default, <1s, $0.006/MP)
- fal-ai/flux-2-pro (previous default, kept backward-compat upscaling)
- fal-ai/z-image/turbo (Tongyi-MAI, bilingual EN/CN)
- fal-ai/nano-banana (Gemini 2.5 Flash Image)
- fal-ai/gpt-image-1.5 (with quality tier: low/medium/high)
- fal-ai/ideogram/v3 (best typography)
- fal-ai/recraft-v3 (vector, brand styles)
- fal-ai/qwen-image (LLM-based)
Architecture:
- FAL_MODELS catalog declares per-model size family, defaults, supports
whitelist, and upscale flag. Three size families handled uniformly:
image_size_preset (flux family), aspect_ratio (nano-banana), and
gpt_literal (gpt-image-1.5).
- _build_fal_payload() translates unified inputs (prompt + aspect_ratio)
into model-specific payloads, merges defaults, applies caller overrides,
wires GPT quality_setting, then filters to the supports whitelist — so
models never receive rejected keys.
- IMAGEGEN_BACKENDS registry in tools_config prepares for future imagegen
providers (Replicate, Stability, etc.); each provider entry tags itself
with imagegen_backend: 'fal' to select the right catalog.
- Upscaler (Clarity) defaults off for new models (preserves <1s value
prop), on for flux-2-pro (backward-compat). Per-model via FAL_MODELS.
Config:
image_gen.model = fal-ai/flux-2/klein/9b (new)
image_gen.quality_setting = medium (new, GPT only)
image_gen.use_gateway = bool (existing)
Agent-facing schema unchanged (prompt + aspect_ratio only) — model
choice is a user-level config decision, not an agent-level arg.
Picker uses curses_radiolist (arrow keys, auto numbered-fallback on
non-TTY). Column-aligned: Model / Speed / Strengths / Price.
Docs: image-generation.md rewritten with the model table and picker
walkthrough. tools-reference, tool-gateway, overview updated to drop
the stale "FLUX 2 Pro" wording.
Tests: 42 new in tests/tools/test_image_generation.py covering catalog
integrity, all 3 size families, supports filter, default merging, GPT
quality wiring, model resolution fallback. 8 new in
tests/hermes_cli/test_tools_config.py for picker wiring (registry,
config writes, GPT quality follow-up prompt, corrupt-config repair).
* feat(image_gen): translate managed-gateway 4xx to actionable error
When the Nous Subscription managed FAL proxy rejects a model with 4xx
(likely portal-side allowlist miss or billing gate), surface a clear
message explaining:
1. The rejected model ID + HTTP status
2. Two remediation paths: set FAL_KEY for direct access, or
pick a different model via `hermes tools`
5xx, connection errors, and direct-FAL errors pass through unchanged
(those have different root causes and reasonable native messages).
Motivation: new FAL models added to this release (flux-2-klein-9b,
z-image-turbo, nano-banana, gpt-image-1.5, ideogram-v3, recraft-v3,
qwen-image) are untested against the Nous Portal proxy. If the portal
allowlists model IDs, users on Nous Subscription will hit cryptic
4xx errors without guidance on how to work around it.
Tests: 8 new cases covering status extraction across httpx/fal error
shapes and 4xx-vs-5xx-vs-ConnectionError translation policy.
Docs: brief note in image-generation.md for Nous subscribers.
Operator action (Nous Portal side): verify that fal-queue-gateway
passes through these 7 new FAL model IDs. If the proxy has an
allowlist, add them; otherwise Nous Subscription users will see the
new translated error and fall back to direct FAL.
* feat(image_gen): pin GPT-Image quality to medium (no user choice)
Previously the tools picker asked a follow-up question for GPT-Image
quality tier (low / medium / high) and persisted the answer to
`image_gen.quality_setting`. This created two problems:
1. Nous Portal billing complexity — the 22x cost spread between tiers
($0.009 low / $0.20 high) forces the gateway to meter per-tier per
user, which the portal team can't easily support at launch.
2. User footgun — anyone picking `high` by mistake burns through
credit ~6x faster than `medium`.
This commit pins quality at medium by baking it into FAL_MODELS
defaults for gpt-image-1.5 and removes all user-facing override paths:
- Removed `_resolve_gpt_quality()` runtime lookup
- Removed `honors_quality_setting` flag on the model entry
- Removed `_configure_gpt_quality_setting()` picker helper
- Removed `_GPT_QUALITY_CHOICES` constant
- Removed the follow-up prompt call in `_configure_imagegen_model()`
- Even if a user manually edits `image_gen.quality_setting` in
config.yaml, no code path reads it — always sends medium.
Tests:
- Replaced TestGptQualitySetting (6 tests) with TestGptQualityPinnedToMedium
(5 tests) — proves medium is baked in, config is ignored, flag is
removed, helper is removed, non-gpt models never get quality.
- Replaced test_picker_with_gpt_image_also_prompts_quality with
test_picker_with_gpt_image_does_not_prompt_quality — proves only 1
picker call fires when gpt-image is selected (no quality follow-up).
Docs updated: image-generation.md replaces the quality-tier table
with a short note explaining the pinning decision.
* docs(image_gen): drop stale 'wires GPT quality tier' line from internals section
Caught in a cleanup sweep after pinning quality to medium. The
"How It Works Internally" walkthrough still described the removed
quality-wiring step.
This commit is contained in:
Teknium
GitHub
parent
0061dca950
commit
01906e99dd
8 changed files with 1245 additions and 468 deletions
|
|
@ -258,14 +258,16 @@ TOOL_CATEGORIES = {
|
|||
"requires_nous_auth": True,
|
||||
"managed_nous_feature": "image_gen",
|
||||
"override_env_vars": ["FAL_KEY"],
|
||||
"imagegen_backend": "fal",
|
||||
},
|
||||
{
|
||||
"name": "FAL.ai",
|
||||
"badge": "paid",
|
||||
"tag": "FLUX 2 Pro with auto-upscaling",
|
||||
"tag": "Pick from flux-2-klein, flux-2-pro, gpt-image, nano-banana, etc.",
|
||||
"env_vars": [
|
||||
{"key": "FAL_KEY", "prompt": "FAL API key", "url": "https://fal.ai/dashboard/keys"},
|
||||
],
|
||||
"imagegen_backend": "fal",
|
||||
},
|
||||
],
|
||||
},
|
||||
|
|
@ -950,6 +952,106 @@ def _detect_active_provider_index(providers: list, config: dict) -> int:
|
|||
return 0
|
||||
|
||||
|
||||
# ─── Image Generation Model Pickers ───────────────────────────────────────────
|
||||
#
|
||||
# IMAGEGEN_BACKENDS is a per-backend catalog. Each entry exposes:
|
||||
# - config_key: top-level config.yaml key for this backend's settings
|
||||
# - model_catalog_fn: returns an OrderedDict-like {model_id: metadata}
|
||||
# - default_model: fallback when nothing is configured
|
||||
#
|
||||
# This prepares for future imagegen backends (Replicate, Stability, etc.):
|
||||
# each new backend registers its own entry; the FAL provider entry in
|
||||
# TOOL_CATEGORIES tags itself with `imagegen_backend: "fal"` to select the
|
||||
# right catalog at picker time.
|
||||
|
||||
|
||||
def _fal_model_catalog():
|
||||
"""Lazy-load the FAL model catalog from the tool module."""
|
||||
from tools.image_generation_tool import FAL_MODELS, DEFAULT_MODEL
|
||||
return FAL_MODELS, DEFAULT_MODEL
|
||||
|
||||
|
||||
IMAGEGEN_BACKENDS = {
|
||||
"fal": {
|
||||
"display": "FAL.ai",
|
||||
"config_key": "image_gen",
|
||||
"catalog_fn": _fal_model_catalog,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _format_imagegen_model_row(model_id: str, meta: dict, widths: dict) -> str:
|
||||
"""Format a single picker row with column-aligned speed / strengths / price."""
|
||||
return (
|
||||
f"{model_id:<{widths['model']}} "
|
||||
f"{meta.get('speed', ''):<{widths['speed']}} "
|
||||
f"{meta.get('strengths', ''):<{widths['strengths']}} "
|
||||
f"{meta.get('price', '')}"
|
||||
)
|
||||
|
||||
|
||||
def _configure_imagegen_model(backend_name: str, config: dict) -> None:
|
||||
"""Prompt the user to pick a model for the given imagegen backend.
|
||||
|
||||
Writes selection to ``config[backend_config_key]["model"]``. Safe to
|
||||
call even when stdin is not a TTY — curses_radiolist falls back to
|
||||
keeping the current selection.
|
||||
"""
|
||||
backend = IMAGEGEN_BACKENDS.get(backend_name)
|
||||
if not backend:
|
||||
return
|
||||
|
||||
catalog, default_model = backend["catalog_fn"]()
|
||||
if not catalog:
|
||||
return
|
||||
|
||||
cfg_key = backend["config_key"]
|
||||
cur_cfg = config.setdefault(cfg_key, {})
|
||||
if not isinstance(cur_cfg, dict):
|
||||
cur_cfg = {}
|
||||
config[cfg_key] = cur_cfg
|
||||
current_model = cur_cfg.get("model") or default_model
|
||||
if current_model not in catalog:
|
||||
current_model = default_model
|
||||
|
||||
model_ids = list(catalog.keys())
|
||||
# Put current model at the top so the cursor lands on it by default.
|
||||
ordered = [current_model] + [m for m in model_ids if m != current_model]
|
||||
|
||||
# Column widths
|
||||
widths = {
|
||||
"model": max(len(m) for m in model_ids),
|
||||
"speed": max((len(catalog[m].get("speed", "")) for m in model_ids), default=6),
|
||||
"strengths": max((len(catalog[m].get("strengths", "")) for m in model_ids), default=0),
|
||||
}
|
||||
|
||||
print()
|
||||
header = (
|
||||
f" {'Model':<{widths['model']}} "
|
||||
f"{'Speed':<{widths['speed']}} "
|
||||
f"{'Strengths':<{widths['strengths']}} "
|
||||
f"Price"
|
||||
)
|
||||
print(color(header, Colors.CYAN))
|
||||
|
||||
rows = []
|
||||
for mid in ordered:
|
||||
row = _format_imagegen_model_row(mid, catalog[mid], widths)
|
||||
if mid == current_model:
|
||||
row += " ← currently in use"
|
||||
rows.append(row)
|
||||
|
||||
idx = _prompt_choice(
|
||||
f" Choose {backend['display']} model:",
|
||||
rows,
|
||||
default=0,
|
||||
)
|
||||
|
||||
chosen = ordered[idx]
|
||||
cur_cfg["model"] = chosen
|
||||
_print_success(f" Model set to: {chosen}")
|
||||
|
||||
|
||||
def _configure_provider(provider: dict, config: dict):
|
||||
"""Configure a single provider - prompt for API keys and set config."""
|
||||
env_vars = provider.get("env_vars", [])
|
||||
|
|
@ -1006,6 +1108,10 @@ def _configure_provider(provider: dict, config: dict):
|
|||
_print_success(f" {provider['name']} - no configuration needed!")
|
||||
if managed_feature:
|
||||
_print_info(" Requests for this tool will be billed to your Nous subscription.")
|
||||
# Imagegen backends prompt for model selection after backend pick.
|
||||
backend = provider.get("imagegen_backend")
|
||||
if backend:
|
||||
_configure_imagegen_model(backend, config)
|
||||
return
|
||||
|
||||
# Prompt for each required env var
|
||||
|
|
@ -1040,6 +1146,10 @@ def _configure_provider(provider: dict, config: dict):
|
|||
|
||||
if all_configured:
|
||||
_print_success(f" {provider['name']} configured!")
|
||||
# Imagegen backends prompt for model selection after env vars are in.
|
||||
backend = provider.get("imagegen_backend")
|
||||
if backend:
|
||||
_configure_imagegen_model(backend, config)
|
||||
|
||||
|
||||
def _configure_simple_requirements(ts_key: str):
|
||||
|
|
@ -1211,6 +1321,10 @@ def _reconfigure_provider(provider: dict, config: dict):
|
|||
_print_success(f" {provider['name']} - no configuration needed!")
|
||||
if managed_feature:
|
||||
_print_info(" Requests for this tool will be billed to your Nous subscription.")
|
||||
# Imagegen backends prompt for model selection on reconfig too.
|
||||
backend = provider.get("imagegen_backend")
|
||||
if backend:
|
||||
_configure_imagegen_model(backend, config)
|
||||
return
|
||||
|
||||
for var in env_vars:
|
||||
|
|
@ -1228,6 +1342,11 @@ def _reconfigure_provider(provider: dict, config: dict):
|
|||
else:
|
||||
_print_info(" Kept current")
|
||||
|
||||
# Imagegen backends prompt for model selection on reconfig too.
|
||||
backend = provider.get("imagegen_backend")
|
||||
if backend:
|
||||
_configure_imagegen_model(backend, config)
|
||||
|
||||
|
||||
def _reconfigure_simple_requirements(ts_key: str):
|
||||
"""Reconfigure simple env var requirements."""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue