Ollama sizes a model's real context window by free VRAM at load time,
often far below the GGUF trained max that /api/show reports, and its
OpenAI-compatible endpoint has no options passthrough — per-request
num_ctx and keep_alive are silently dropped, so the window cannot be
controlled from the client. The compressor was being sized to the
trained max (e.g. 262K for a model actually running at 32K).
- Add query_ollama_loaded_context() reading the effective window from
/api/ps (60s cache, never persisted — transient load state).
- Reconcile after each successful response via
sync_ollama_loaded_context(): resize the compressor to the loaded
window and warn when it is below the tool-use minimum. Selection
surfaces keep showing the trained max; explicit model.context_length
still wins. No-op for non-Ollama providers.
- Refresh model.ollama_keep_alive through the native API (rate-limited
/api/generate ping) since /v1 drops it.
- Remove the inert num_ctx/keep_alive request-body plumbing; warn that
model.ollama_num_ctx has no effect and point at OLLAMA_CONTEXT_LENGTH
/ Modelfile num_ctx. Keep detection for the pre-flight window check.
- Disable thinking on Ollama thinking models via reasoning_effort
'none' — the only switch its /v1 handler parses (think is dropped).
Promote bare "ollama" from a custom-endpoint alias to a real provider and
build the desktop UX around it. A local server's connection kind is
reachability rather than a credential, so every credential-shaped gate
(provider registry, picker filters, settings surfaces) gets an explicit
path for it.
Backend:
- Provider overlay + registry entry (127.0.0.1:11434/v1 default, keyless
with a local-only placeholder, base-url normalization for /api and /v1
forms). Existing provider=custom configs are untouched.
- GET /api/local-servers/detect fingerprints well-known local ports plus a
local configured base_url; response shape leaves room for a future
installed/running/managed distinction.
- /api/ollama/* management endpoints: installed+running+recommended models,
registry pull as a poll-able background job streaming native NDJSON
progress, delete, and load (warm-up / keep_alive pinning). Pull and
delete bust the picker's model-id cache.
- Model picker payload: per-model capabilities widened to tools/vision/
context_length; local Ollama rows enriched from the server's native
/api/show (authoritative for on-disk tags, where models.dev is sparse),
backfilled off the request path by a background thread.
- The explicit-only picker filter keeps ollama rows: the row only exists
when the server answered a probe, which is as explicit as a pasted key.
- Reasoning safety: /api/show thinking capability gates all reasoning
fields (Ollama 400s reasoning_effort on non-thinking models), and
OpenAI-only effort levels map to the nearest accepted level
(xhigh->max, minimal->low).
- model.ollama_keep_alive config: sent per-request as extra_body.keep_alive.
- Latency discipline for a local server that may be down: a 300ms TCP
pre-check with a short negative cache guards every native-API read; the
status endpoint probes fresh so a just-started server is noticed
immediately; localhost is rewritten to 127.0.0.1 (Windows resolves
localhost to ::1 first and Ollama binds IPv4 loopback — each request
otherwise pays a ~2s failed IPv6 connect, including chat inference).
Desktop:
- Providers -> Accounts: a "Local servers" card mirroring the OAuth card
language ("Running · N models" / a start-the-server hint), expanding to
model management: installed models with size/quant/VRAM, delete, warm-up,
curated pull recommendations with a progress bar, free-form pull, and a
KV-cache advisory when a loaded model runs well under its trained window.
Polls while down so it flips to Running by itself.
- Model picker: "No tools" badge (explicit tools:false only — absence means
unknown) with demotion, plus context window / parameter size / quant per
row; refetches once after open so backfilled metadata appears in place.
- Onboarding: detected-server row with a model select, replacing blind
first-model assignment for detected servers.
- Composer status stack: "Loading <model> into memory" row during cold
starts, confirmed against /api/ps so ordinary slow generations stay quiet.
Chat inference stays on the OpenAI-compatible /v1 endpoint; the native
/api surface is used read-only for metadata plus explicit management
actions. Lifecycle management (starting or installing Ollama) is not
included.
PR #57601's original branch added a top-level reasoning_effort emit to the
LEGACY build_kwargs path (agent/transports/chat_completions.py), but
provider=custom resolves to CustomProfile (plugins/model-providers/custom/),
so chat_completion_helpers takes the profile path and returns early — the
added branch was unreachable dead code for every custom endpoint.
Move the fix to its real site, CustomProfile.build_api_kwargs_extras(), and
follow the DeepSeek/Zai profile precedent:
- disabled -> extra_body.think = False (unchanged)
- enabled + effort -> TOP-LEVEL reasoning_effort (the OpenAI-compatible
format GLM-5.2/ARK expect), passed through verbatim
incl. max/xhigh
- enabled + no effort -> omit, so the endpoint's server default applies
(avoids silently forcing 'medium' as the original
branch did)
Deliberately does NOT force think=True on enable — that flag is Ollama-only
and risks a 400 on GLM/vLLM endpoints that don't recognize it; thinking is
already server-default-on for these backends.
Verified end-to-end through the real profile dispatch (temp HERMES_HOME):
custom+high -> reasoning_effort=high; custom+max -> reasoning_effort=max;
custom+none -> think=False; custom+unset -> nothing; num_ctx composes.
Adds tests/plugins/model_providers/test_custom_profile.py (13 cases).
Addresses the custom-provider half of #55276.
Co-authored-by: huanshan5195 <huanshan5195@users.noreply.github.com>
The /model interactive picker resolved a base_url from user credentials
but never passed it to ProviderProfile.fetch_models(), causing the
picker to always query the provider's hardcoded default endpoint
instead of the user's custom URL (e.g. a company litellm proxy).
- providers/base.py: add optional base_url parameter to fetch_models()
- hermes_cli/models.py: pass resolved base_url to fetch_models()
- Update all subclass overrides for signature compatibility
- Add 6 regression tests covering override, fallback, and integration
The custom/Ollama provider profile had no default_max_tokens, so no
max_tokens was sent on requests and Ollama fell back to its internal
num_predict=128 — truncating responses after a few tokens with
finish_reason='length' (#39281, e.g. gemma4).
max_tokens resolution is ephemeral > user model.max_tokens > profile
default, so this is only a floor used when the user hasn't set their own
cap. Set it to 65536 (matching the qwen-oauth tier) rather than a
conservative value, since users can always override per-model.
Fixes#39281
Every provider profile is now a self-contained plugin under
plugins/model-providers/<name>/, mirroring the plugins/platforms/
pattern established for IRC and Teams. The ProviderProfile ABC
stays in providers/; the per-provider profile data moves out.
- plugins/model-providers/<name>/__init__.py calls register_provider()
- plugins/model-providers/<name>/plugin.yaml declares kind: model-provider
- providers/__init__.py._discover_providers() lazily scans bundled plugins
then $HERMES_HOME/plugins/model-providers/<name>/ (user override path)
- User plugins with the same name override bundled ones (last-writer-wins
in register_provider)
- Legacy providers/<name>.py layout still supported for back-compat with
out-of-tree editable installs
- Hermes PluginManager: new kind=model-provider; skipped like memory
plugins (providers/ discovery owns them); standalone plugins with
register_provider+ProviderProfile in their __init__.py auto-coerce to
this kind (same heuristic as memory providers)
- skip_names extended to include 'model-providers' so the general
PluginManager doesn't double-scan the category
- 4 new tests in tests/providers/test_plugin_discovery.py covering
bundled discovery, user override, and general-loader isolation
- Docs updated: website/docs/developer-guide/adding-providers.md,
provider-runtime.md, providers/README.md, plugins/model-providers/README.md
No API break: auth.py / config.py / doctor.py / models.py / runtime_provider.py /
model_metadata.py / auxiliary_client.py / chat_completions.py / run_agent.py
all still consume providers via get_provider_profile() / list_providers() —
they just now see plugin-discovered entries instead of pkgutil-iterated ones.
Third parties can now drop a single directory into
~/.hermes/plugins/model-providers/<name>/ to add or override an inference
provider without touching the repo.
