Commit graph

2255 commits

Author SHA1 Message Date
Victor Kyriazakos
efbf2fd79c feat(monitoring): add cron operational telemetry 2026-07-24 18:54:46 +00:00
Victor Kyriazakos
9b098e7f78 fix(monitoring): enrich gateway diagnostic scope 2026-07-24 18:54:46 +00:00
Victor Kyriazakos
42bc6c8626 fix(monitoring): keep diagnostics content-free 2026-07-24 18:54:46 +00:00
Victor Kyriazakos
18b4664543 fix(monitoring): drain terminal lifecycle events 2026-07-24 18:54:45 +00:00
Victor Kyriazakos
73190cdbfe fix(monitoring): complete production OTLP runtime 2026-07-24 18:54:45 +00:00
Victor Kyriazakos
98a0260b0a fix(otel): identify Hermes resources safely 2026-07-24 18:54:45 +00:00
Victor Kyriazakos
16e774f223 fix(monitoring): harden gateway health OTLP egress 2026-07-24 18:54:45 +00:00
Victor Kyriazakos
87a15733c0 fix(monitoring): address review — persist install_id, drop dead config, single redaction path
- Cut the leftover telemetry.* DEFAULT_CONFIG block (nothing reads it) and
  the legacy telemetry-key fallback in policy.py.
- install_id: persist the minted UUID back to config.yaml on first use so
  service.instance.id survives gateway restarts (fail-open when the write
  is not possible); regression test covers the restart path.
- Remove the no-op gateway_health_export.redaction config keys. Redaction
  is always-on by design and deliberately not configurable; status output
  now says so.
- Collapse redaction to one unconditional secrets+PII scrub: drop the
  none/pii content modes (they served the dropped trajectories plane) and
  fold gateway_health.py's duplicate bearer/token/email/phone regex layer
  into agent/monitoring/redaction.py.
2026-07-24 18:54:45 +00:00
Victor Kyriazakos
505d12f662 refactor(monitoring): scope telemetry substrate to gateway health/diagnostics export
Salvages the event-spine foundation from feat/telemetry-observability
(emitter, typed events, OTLP streaming, redaction — authorship preserved in
the preceding commits) and scopes it to the plane enterprise operators need
today: gateway Service Health Monitoring plus redacted Operational
Diagnostics, exported over OTLP.

Dropped from the salvaged branch, deliberately:
- run/model/tool trajectory capture (plugins/telemetry hooks, tel_spans)
- the local JSONL + state.db tel_* store (monitoring is egress, not storage)
- usage rollups/metrics, /insights integration, bulk export
- hermes telemetry CLI (replaced by hermes monitoring status)

Those planes — shared client usage metrics and enterprise trace telemetry —
are being designed on the NeMo Relay integration with distinct consent,
policy, and export boundaries; this keeps the monitoring plane content-free
and independently enableable.

Renames agent/telemetry -> agent/monitoring, config telemetry.* ->
monitoring.*, and pins the otlp extra at OpenTelemetry 1.39.1 (matching
uv.lock; 1.30.0 conflicts with mistralai>=2.4 on opentelemetry-api).
2026-07-24 18:54:45 +00:00
emozilla
edac3a7bae refactor(telemetry): cut dead schema; tests assert what's actually written
Self-review after the #51714 feedback found the reviewer's dead-table finding
was not isolated — the schema advertised far more than the code populates, and
our own tests hid it by hand-feeding fields production never sends. Make the
surface honest by subtraction.

Schema (10 tel_* tables -> 5):
  - Delete tel_gateway_events, tel_cron_events, tel_skill_events,
    tel_memory_events, tel_feedback_events — declared, never written, never read.
  - Drop columns nothing populates: tel_runs.{profile_id,estimated_cost_usd,
    cost_status}; tel_model_calls.{ttft_ms,estimated_cost_usd,cost_status,
    cost_source,end_reason,retry_count}; tel_tool_calls.{backend,retry_count,
    approval}; tel_spans.attrs_json. Cost duplicated the existing sessions
    billing columns and was always NULL here.
  - events.py / emitter _TABLE_COLUMNS / OTLP _span_attrs / rollup / preview
    display all trimmed to match.

Correctness:
  - end_reason no longer hardcodes "completed". Production finalize callers pass
    `reason` (shutdown/session_expired/session_reset); _coarse_end_reason now
    reads it and maps accordingly.
  - Fix a latent bug the trim exposed: the model_call hook passed end_reason= to
    ModelCallEvent, which the @_safe wrapper was silently swallowing — so
    tel_model_calls dropped every row in real runs. Now writes correctly.

Tests:
  - Stop hand-feeding estimated_cost_usd / turn_exit_reason that no production
    call site sends. Finalize is now driven with the real `reason` kwarg, and
    assertions cover only fields that are actually populated. This is what let
    the model_call drop hide — the suite graded on a fictional contract.

Net: a smaller system that does what it says. Verified end-to-end over the real
dispatch path (runs + connected span tree + model/tool rows populate; dead
tables gone). 160 telemetry/state/insights tests green.
2026-07-24 18:54:45 +00:00
emozilla
7c80b79bb0 feat(telemetry): write tel_spans — reconstructable run -> calls trace
Addresses the review on #51714: the trace/span layer was declared but unwired —
tel_spans was never written, call rows had no timestamp, and nothing set parent
lineage, so the store was metrics-only and couldn't reconstruct a trace.

Wire the span layer (keeping the praised star-schema shape):
  - New SpanEvent (span_id/trace_id/run_id/parent_span_id/name/kind/start_ns/end_ns)
    mapped into tel_spans via the emitter's _TABLE_COLUMNS.
  - The plugin mints a root span per run and, on each model/tool call, emits a
    SpanEvent (timing + parent = the run's root) keyed by the SAME span_id as the
    detail row, so tel_model_calls / tel_tool_calls JOIN to their span.
  - Call hooks fire on completion, so end_ns = now and start_ns is reconstructed
    from the measured latency/duration. The run's root span is emitted at finalize
    with the true run start/end.

Result: tel_spans is a connected, single-trace_id, run -> calls tree a desktop
waterfall (or any reader) can render directly, ordered by start_ns. Existing
metrics rows (tel_runs/model_calls/tool_calls) are unchanged.

OTLP: spans now flow to the exporter with their trace/parent/timing attributes.
The exporter still emits one OTel span per event rather than reconstructing OTel
SpanContexts into a connected trace tree; that projection is left for a follow-up
and the module docstring now says so plainly instead of over-claiming.

Adds test_spans_trace.py (connected-tree + detail-row JOIN) over the real dispatch
path. Accurate (pre-hook) start times, real OTLP SpanContexts, and subagent
cross-run lineage remain follow-ups.
2026-07-24 18:54:45 +00:00
emozilla
24c86e160b fix(telemetry): aggregate requires local telemetry to be on
Aggregate metrics are derived from the local tel_* tables — they're a coarsened
view of local data, not an independent capture path. With telemetry.local=false
nothing is written, so an aggregate opt-in had nothing to aggregate, yet
may_upload_aggregate() returned True and `status` showed "Aggregate metrics: on".
The config could claim a state it couldn't fulfill.

Gate aggregate on local being enabled:
  - may_upload_aggregate() now requires local_enabled AND allow_aggregate AND
    consent_state == aggregate.
  - `telemetry status` computes aggregate_enabled the same way and, when consent is
    aggregate but local is off, prints "inert: local telemetry is off — nothing to
    aggregate" instead of the opt-in hint.

Happy path is unchanged (local on + consent aggregate -> on). Adds policy and CLI
tests for the inert combo.
2026-07-24 18:54:45 +00:00
emozilla
72b7af73bd refactor(telemetry): drop "plane" terminology
Rename the telemetry tiers away from the borrowed control-plane/data-plane
jargon to plain language, across code, CLI output, config, and docs:

  - "local plane"        -> "local telemetry"
  - "aggregate plane"    -> "aggregate metrics"
  - "trajectories plane" -> "trajectories" / "telemetry.trajectories"
  - "three planes with a hard wall" -> "three settings, isolated from each other"

User-facing `hermes telemetry status` now reads "Local telemetry: on" /
"Aggregate metrics: off" / "Content export: off (trajectories disabled)".
The OTLP resource attribute key telemetry.plane is renamed to telemetry.scope
(wire-level identifier; nothing consumes it yet).

No behavior change — wording only. Status renders identically apart from the
labels; tests updated to match the new strings.
2026-07-24 18:54:45 +00:00
emozilla
e829d03fec refactor(telemetry): drop policy.resolve(); read config directly
policy.resolve() / TelemetryDecision was a read-only projection used only by
`hermes telemetry status` for display. The actual behavior gates already read
telemetry.* straight from config: the emitter (whether to write) and the plugin
loader (whether to auto-load) each call .get("local", True) on the loaded config,
never through policy.

Make config the single chokepoint the status command reads too: it now resolves
local/allow_aggregate/consent_state inline from the loaded config, the same way
the other gates do. policy.py keeps only what config can't express on its own —
the consent constants, ensure_install_id(), and may_upload_aggregate(config) as a
pure function (the gate a future uploader must consult). resolve() and the
TelemetryDecision dataclass are removed; policy.py drops 107 -> 70 lines.

No behavior change: status renders identically, and the default-on local plane is
still defaulted in DEFAULT_CONFIG plus a fail-safe .get(..., True) at each gate.
2026-07-24 18:54:45 +00:00
emozilla
3e28eaccde feat(telemetry): local-first telemetry & observability
Add a built-in telemetry system that records what the agent does — workflows,
model calls, tool calls, errors — to the local machine, powers `/insights`, and
can export to an operator-chosen destination. Default-on locally; nothing leaves
the machine unless the user exports it or opts into the aggregate plane.

Three planes with a hard wall between them:
  - local: full-fidelity observability (real model/provider/tool names), on by
    default, never leaves the machine.
  - aggregate: opt-in metadata, default off. No uploader ships — consent is
    recorded via telemetry.consent_state, and `preview` shows what would be
    produced, computed locally.
  - trajectories: full message content, opt-in, exported only to the operator's
    own destination.

Mechanism:
  - Bundled `telemetry` plugin registers observational lifecycle hooks
    (on_session_start / post_api_request / post_tool_call / on_session_finalize).
    No core call sites are edited; hooks already carry the data.
  - Fire-and-forget emitter: emit() returns in microseconds, never blocks or
    raises into a model/tool call. A daemon thread writes events to an
    append-only JSONL log and the tel_* tables in state.db (its own sqlite
    connection, separate from SessionDB).
  - tel_runs / tel_model_calls / tel_tool_calls live in the declarative
    SCHEMA_SQL and are reconciled automatically; SCHEMA_VERSION 16 -> 17.
  - metrics derives rollups for /usage and /insights; rollup builds per-run
    summaries for `hermes telemetry preview`.

Consent is config, not a parallel command surface. The config file is the root
of trust: set telemetry.consent_state with `hermes config set`, or pin any
telemetry.* key (including allow_aggregate) via managed scope, which overrides
the user's value per key. `hermes telemetry` exposes only what config cannot:
status (report), preview (query), and export.

Export:
  - exporter_bulk writes telemetry (and, when the trajectories plane is enabled,
    session content) to ndjson/json.
  - otlp_exporter streams spans to a configured OpenTelemetry Collector over
    OTLP/HTTP. The SDK is an optional extra (hermes-agent[otlp]), lazily
    installed via tools.lazy_deps on first use.
  - Secrets are always redacted on every export path
    (redact_sensitive_text(force=True)); content export is gated by the
    trajectories plane, and PII scrubbing follows telemetry.content_redaction.
    OTLP auth headers reference environment variable names, never inline values.

No outbound emission to Nous. The aggregate uploader is intentionally not built.
2026-07-24 18:54:45 +00:00
teknium1
d4b867cf9f fix(windows): sweep remaining unguarded text-mode subprocess sites codebase-wide
AST-driven pass over every subprocess.run/Popen/check_output/check_call/call
with text=True (or universal_newlines=True) and no explicit encoding=:
append encoding='utf-8', errors='replace' at the kwarg site. 136 call
sites across 28 files (cli.py, hermes_cli/main.py, tools_config.py,
environments, computer_use, gateway, scripts, skills helpers, agent/*).

Together with the salvaged #55339/#60741 commits this closes out issue
#53428's bug class; the salvaged #60751 linter rule in
check-windows-footguns.py now enforces it repo-wide (verified: 807 files
scanned, zero findings).
2026-07-24 11:45:57 -07:00
Stoltemberg
c89481db5e fix: add explicit UTF-8 encoding to all subprocess text=True calls (#53428)
On Windows with Chinese locale (GBK), subprocess.run(text=True) without
explicit encoding causes UnicodeDecodeError crashes. This fix adds
encoding='utf-8', errors='replace' to all subprocess.run() and
subprocess.Popen() calls that use text=True across 76 non-test Python files.

Fixes #53428 (master tracker for Windows GBK locale crash).

Note: credential_pool.py and electron changes excluded per reviewer request —
those will be submitted as separate focused PRs.
2026-07-24 11:45:57 -07:00
xxxigm
a9e7b32162 fix(fallback): allow xai-oauth → xai failover on shared host/model
Base-url+model dedup was meant for custom shim aliases, but it also
skipped first-class providers that share an inference host while using
different credentials. That stranded xai-oauth spending-limit failover
to the xai API-key provider when both used the same model slug.
2026-07-24 23:22:11 +05:30
kshitij
c769081803 refactor: extract sync_credential_pool_entry_id helper
Replace 3 duplicated entry_id resolution blocks (try/except +
entry_id_for_api_key + fallback to None) in agent_init.py,
chat_completion_helpers.py, and switch_model with a single
sync_credential_pool_entry_id(agent) function in agent_runtime_helpers.

Follow-up to #70323.
2026-07-24 22:58:47 +05:30
Gille
73c4b5a045 fix(auth): stop stale-key credential recovery loops
Track the selected credential by stable pool entry ID so token refreshes and shared cursor movement cannot detach failures from the entry that issued them. Stop unmatched single-entry pools from reporting a no-op rotation as successful recovery.

Co-authored-by: Maxim Esipov <maksesipov@gmail.com>
2026-07-24 22:58:47 +05:30
Brooklyn Nicholson
a96f7c805b feat(i18n): add Arabic (ar) catalog for agent/CLI messages
Registers `ar` in the supported-language set and alias table and ships
locales/ar.yaml at full key and placeholder parity with en.yaml, covering
approval prompts and gateway slash-command replies. Identifiers, commands,
paths, config keys, model/provider names, and {placeholder} tokens are kept
verbatim.

Co-authored-by: Da7-Tech <286182457+Da7-Tech@users.noreply.github.com>
2026-07-24 12:10:03 -05:00
teknium1
397e9fc1e4 Reapply "Merge pull request #30179 from NousResearch/feat/iron-proxy"
This reverts commit c6dc7c03c3.
2026-07-24 09:49:00 -07:00
brooklyn!
80e575dfba
Merge pull request #70604 from NousResearch/bb/profile-routing-super
fix(sessions): keep a conversation on its owning profile through branch and compression
2026-07-24 10:11:10 -05:00
flyingdoubleg
e9a7c18890 fix(memory): honor disabled toolsets for provider tools 2026-07-24 13:00:53 +05:30
Brooklyn Nicholson
e9a243ef78 fix(state): inherit and stamp profile_name across rotation and branch children
profile_name was only written on the agent's initial lazy create
(e8b7ce8c1); every parented child row — compression rotation, TUI
/branch, desktop branch first-persist — was created without it. A
non-default profile's lineage therefore turned NULL on its first
compression or branch and aggregated as "default" in unified session
lists, completing the cross-profile session-jump.

Fix the class at the DB layer: _insert_session_row's parent backfill now
COALESCEs profile_name from the parent alongside cwd/git_* (#64709
pattern), so any parented child inherits its lineage's owning profile.
Stamp it explicitly at the three create sites as well — compression
rotation (mirroring _ensure_db_session), TUI session.branch, and the
TUI first-prompt row persist — so rows are self-describing even when the
parent row predates the profile_name column.
2026-07-24 01:49:22 -05:00
Teknium
23476207bc feat(moa): default advisor fanout to user_turn — the cheapest cadence
Flips the default fan-out cadence from per_iteration (advisors re-run on
every tool iteration, multiplying advisor spend by tool-loop depth) to
user_turn (advisors run once on the first message of each user turn; the
acting aggregator works the rest of the tool loop with that turn's
advice). Until per-mode benchmarks justify a costlier default, MoA
defaults to the cheapest, lowest-impact cadence (#67199).

One default for everyone — no split legacy/new-preset semantics; presets
that want per-step advising set fanout: per_iteration explicitly. All
three modes (user_turn / per_iteration / every_n:N) remain selectable;
every_n:1 still collapses to per_iteration (semantic identity), while
unparseable values now fall to user_turn (the default).

Docs updated with a default-change note; the per-iteration rerun test
pins its mode explicitly.

Co-authored-by: skyer-flyyy <188930297+skyer-flyyy@users.noreply.github.com>
2026-07-23 21:07:18 -07:00
Alan Harman-Box
bc744d30e0 docs: document 57-char system prompt truncation in authoring guide and curator
The skill-authoring guide and curator prompt both reference
descriptions as the primary discovery mechanism but never mentioned
the 57-char system prompt truncation. Add explicit guidance:

- Authoring guide: frontmatter docs, template comment, size limits,
  pitfall #3 with good/bad examples, verification checklist
- Curator prompt: parenthetical noting the 57-char window when
  writing umbrella skill descriptions
2026-07-23 21:06:56 -07:00
Alan Harman-Box
accbf4d912 feat: show system_prompt_preview when skill description exceeds prompt limit
When a skill is created or edited with a description longer than
SKILL_PROMPT_DESC_LIMIT (60 chars), the tool response now includes a
system_prompt_preview field showing exactly what the system prompt
skill index will display. This gives the agent immediate feedback to
self-correct truncated trigger phrases.

Also adds tool schema guidance about the 57-char window and fixes a
stale docstring in skill_commands.py that incorrectly claimed the
system prompt renders the full description.
2026-07-23 21:06:56 -07:00
Alan Harman-Box
5eb772111d refactor: extract SKILL_PROMPT_DESC_LIMIT constant and normalize description helpers
The system prompt skill index truncates long descriptions to 57 chars,
but this limit was a hardcoded magic number. Extract it as a named
constant and factor the normalization logic into a shared private
helper so the extraction function and the new truncation predicate
cannot drift.

No behaviour change — pure refactor.
2026-07-23 21:06:56 -07:00
Teknium
13fe08d7e9 fix(context-engine): short-circuit the inherited no-op select_context before any per-request work
Verification follow-up for the #51226 salvage: the host call site guarded
select_context with hasattr(), but the ABC defines a default on every
engine, so the built-in ContextCompressor (and any non-implementing
engine) still paid per-request shallow copies of the conversation
history plus a hook call on every provider request. Identity-check the
bound method against ContextEngine.select_context and return the
request untouched — mirroring the existing base-method short-circuit in
_notify_context_engine_turn_complete — so the default path does zero
work, not just produces an identical result.

Adds two pins: the base no-op is never invoked (patched-to-raise base
stays silent), and ContextCompressor.__dict__ contains neither new verb.

Also registers the contributor email mapping for @chaos-xxl.
2026-07-23 19:44:35 -07:00
xue xinglong
5f65f0b0f8 fix(context-engine): snapshot select_context read-only inputs; scope on_turn_complete coverage doc
Addresses the hermes-sweeper review on #51226:
- _apply_context_engine_selection now passes shallow copies of the read-only
  conversation_messages / incoming_message to the hook, so an engine mutating
  them in place cannot corrupt persisted transcript state (enforces the
  request-only contract, not just documents it). Adds a mutation-regression
  test asserting persisted history + incoming message are untouched.
- on_turn_complete docstring: scope the coverage claim to the standard
  finalization seam. Some abnormal early-return paths (content-policy block,
  provider terminal failure) currently persist+return without finalization and
  don't emit the hook; documented as best-effort with a shared-seam follow-up,
  rather than over-promising a guaranteed callback for every early exit.
2026-07-23 19:44:35 -07:00
xue xinglong
915942935d fix(context-engine): fail open on empty select_context() result + doc public hooks
- _apply_context_engine_selection: reject an empty list. all([]) is True, so
  a [] returned by a failing/buggy engine previously replaced a valid request
  with an empty message list the downstream sanitizers can't restore; now it
  falls open to the unmodified request (honors the fail-open contract).
  Thanks @johnnykor82 for catching this on #41918's review.
- test: empty list keeps the original request (fail-open regression).
- docs: document select_context()/on_turn_complete() in the public
  context-engine plugin guide (were still describing only the old contract).
2026-07-23 19:44:35 -07:00
xue xinglong
71220cdf5b docs+test(context-engine): document select_context ordering/cache contract; add cache-stability + downstream-sanitizer tests
- context_engine.py: document that select_context() runs before cache-control
  and all request sanitizers, so (a) replacements still pass host validation
  and (b) the no-op default keeps the request byte-stable (AGENTS.md prompt-
  cache invariant). Note the hook is evaluated per provider request.
- tests: no-op path is byte-stable for cache-control; a role-unusual
  replacement is passed through for the existing downstream sanitizers to
  normalize (select_context does structural validation only).
2026-07-23 19:44:35 -07:00
xue xinglong
589cbafb87 feat(context-engine): forward real usage to on_turn_complete()
The on_turn_complete() observation hook is the engine's post-turn signal,
so it should receive the completed turn's canonical token usage when the
host has it, not a hardcoded None. Per @johnnykor82's #41918 contract: the
engine uses prompt/completion + cache_read/write/reasoning buckets to judge
how large/expensive the selected context was before the next select_context().

- conversation_loop.py: stash the most recent provider response's usage_dict
  (the same canonical shape fed to update_from_response) on the agent as
  _last_turn_usage; reset to None at turn start so turns that never reach a
  provider response (early failure / interrupt) forward None, not a stale
  prior turn's usage.
- turn_finalizer.py: forward agent._last_turn_usage instead of usage=None.
- context_engine.py: document the usage param contract on the ABC hook.
- tests: cover both ends through the real finalize_turn path — completed turn
  forwards the full canonical bucket set intact; no-response turn forwards None.

Co-authored-by: johnnykor82 <johnnykor82@users.noreply.github.com>
2026-07-23 19:44:35 -07:00
xue xinglong
bb9ef9d72c feat(context-engine): add on_turn_complete() observation hook
Adds the post-turn observation verb as the companion to select_context():
an optional, no-op-default on_turn_complete() called once after the
assistant/tool loop finishes, with the finalized transcript snapshot. Lets
an engine ingest/index/summarize the completed turn to inform the next
select_context(). Wired via _notify_context_engine_turn_complete() from
turn_finalizer.finalize_turn(); fail-open, base no-op short-circuited so
non-implementing engines (incl. the built-in compressor) pay nothing.

This is the request-assembly + observation pair from #41918; with this
commit the PR fully subsumes #41918's two hooks (prepare_request_messages
-> select_context, on_turn_complete) rather than only the selection half.

Co-authored-by: johnnykor82 <johnnykor82@users.noreply.github.com>
2026-07-23 19:44:35 -07:00
xue xinglong
dec464c351 feat(context-engine): add select_context() per-turn selection hook
Adds an optional, no-op-default select_context() hook to the ContextEngine
ABC, called every turn after the request messages are assembled and before
provider dispatch — independent of should_compress(). Lets an engine select
or replace which context enters the prompt for a single request (retrieval,
topic routing, role/branch switching) without mutating persisted history,
removing the need to abuse should_compress()=True as a per-turn callback.

The host call site (_apply_context_engine_selection) is fail-open: a missing
hook, an exception, or an invalid return value leaves the assembled request
untouched. Additive and non-breaking: the built-in compressor and every
existing engine are unaffected.

Consolidates the per-turn request-assembly surface proposed across #41918,

Related: #36765 #41918 #24949 #47109 #50053 #23837 #25115 #29370
2026-07-23 19:44:35 -07:00
Teknium
7b65073dc9 fix(moa): tolerate SDK-shaped tool_call entries in _render_tool_calls
_render_tool_calls only handled dict-shaped entries; a SimpleNamespace-
shaped tool_call (SDK-style stream-stitched responses) rendered as
'[called tool: tool]', silently losing the function name and arguments
from the advisory view. Handle both shapes (including a namespace-shaped
nested function inside a dict entry).

One-hunk hardening salvaged from closed #59712.

Co-authored-by: SquabbyZ <601709253@qq.com>
2026-07-23 18:40:09 -07:00
Teknium
975eb3a365 fix(moa): trim reference messages to fit each model's context window
Reference models may have a smaller context window than the aggregator
(e.g. kimi-k2.7-code @ 262K advising a glm-5.2 @ 1M conversation).
Without context-length protection, a reference whose window is exceeded
gets a hard HTTP 400 from the provider, which _run_reference's
try/except silently converts to a [failed: …] note — the MoA turn
silently degrades to fewer references (#60345).

Redesigned implementation of #60387:
- Estimate AFTER the advisory system prompt is prepended, so the
  request that is actually sent is what gets budgeted.
- Reserve output headroom: the preset's reference_max_tokens when set,
  else an 8192-token constant, plus a 10% estimator-error fraction.
- Trim on advisory-view boundaries (text-only user/assistant turns; no
  tool-result frames to orphan), preserving the system prompt, the
  user-first invariant after every pop (never assistant-first), and the
  trailing synthetic user turn.
- Cache get_model_context_length per (provider, model) in a per-fan-out
  dict shared across the worker threads, so a turn resolves each
  window once instead of probing metadata sources
  per-reference-per-iteration (failures are cached too).

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
2026-07-23 18:40:09 -07:00
Teknium
55f3826224 fix(moa): keep real accounting for interrupted-but-billed references; don't cache interrupted results
Follow-ups for salvaged #56344:

- A reference that completes between the interrupt check and the reap
  keeps its REAL output and accounting (the provider call billed) instead
  of being zeroed with a placeholder.
- A reference still in flight at interrupt time gets a placeholder in the
  results, but its future now carries a done-callback that folds the
  eventual real usage/cost into the facade's pending accounting
  (late_accounting_sink -> _record_late_reference_accounting), so billed
  spend is never silently dropped. Pending totals are folded (not
  overwritten) and guarded by a lock since done-callbacks fire on
  executor worker threads.
- Interrupted placeholder results are no longer written into the facade's
  turn-scoped reference cache: a cache HIT never re-runs references, so
  caching a partial snapshot would replay '[skipped: interrupted by
  user]' notes for the rest of the turn. The cache is left empty and the
  next create() re-runs the fan-out.
2026-07-23 18:40:09 -07:00
srojk34
68cd755731 fix(moa): allow a user interrupt to abort the reference fan-out wait
agent/tool_executor.py's concurrent tool batch checks agent._interrupt_requested
and aborts the wait early; agent/moa_loop.py's _run_references_parallel had
no equivalent, so a MoA-enabled turn blocked on ThreadPoolExecutor.result()
until every reference model finished or hit its own individual
auxiliary.moa_reference timeout -- there was no way for the user to abort a
live turn mid-fanout.

Thread an optional `agent` parameter through aggregate_moa_context ->
_run_references_parallel (used when MoA references run alongside the main
model) and MoAClient/MoAChatCompletions (used when the MoA preset itself is
the acting model), then poll concurrent.futures.wait() in
_REFERENCE_POLL_INTERVAL_S slices instead of blocking on future.result() per
reference, checking agent._interrupt_requested each cycle.

Deliberately scoped to interrupt/cancel only -- no new or changed timeout
value, so this doesn't overlap open PRs #53784/#53875 (which lower the
per-reference timeout default but don't add interrupt support). `agent` is
optional and defaults to None, so any caller that doesn't pass it keeps
today's uninterruptible blocking behavior unchanged.
2026-07-23 18:40:09 -07:00
Teknium
62c2b299a3 fix(moa): act aggregator-alone on the facade path when all references fail
Extends the all-references-failed short-circuit (#56975) to the
persistent `provider: moa` facade path: MoAChatCompletions.create()
previously attached 'use the reference responses below' guidance built
entirely from failure sentinels and called the aggregator with it. Now
an all-failed turn attaches either the sanitized unavailability notice
(loud policy) or nothing (silent policy), and the aggregator — which IS
the acting model — simply acts alone. Advisor accounting for the failed
fan-out is still recorded.

Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
2026-07-23 18:40:09 -07:00
liuhao1024
f0ed77b627 fix(moa): skip aggregator synthesis when all references fail
When every MoA reference model returns a failure (HTTP error, timeout,
etc.) or is skipped by the recursion guard, the one-shot aggregator
synthesis call is now skipped entirely. Previously it would try to
synthesise a wall of failure sentinels, which could block for the full
provider timeout (observed ~6 min on SenseNova) before returning a
non-retryable error that left the session hanging.

The early return carries the sanitized unavailability notice (never raw
provider error text, per the failed-reference containment) so the main
agent loop can still act in single-model mode.

Salvaged from #56975, reworked atop the _is_failed_reference helpers.
2026-07-23 18:40:09 -07:00
Teknium
d3fc27bbf8 fix(moa): make reference_timeout default inherit auxiliary config; filter recursion-guard skips
Follow-ups for salvaged #53784:

- reference_timeout now defaults to None = no per-preset override, so the
  reference fan-out inherits auxiliary.moa_reference.timeout (900s default)
  via call_llm's own per-task timeout resolution. The PR's 30.0s default
  would have cut off long-thinking advisors mid-response, and its 300s max
  cap capped legitimate explicit values — both removed. Explicit per-preset
  values are still honored as-is.
- _is_failed_reference also treats '[skipped: …]' recursion-guard notes as
  internal sentinels, keeping them out of both aggregator prompts.
- Dashboard/desktop TS types updated to number | null; web_server validator
  accepts null/empty as 'inherit'.
2026-07-23 18:40:09 -07:00
robbyczgw-cla
ccdf171bcd fix(moa): contain failed reference details 2026-07-23 18:40:09 -07:00
liuhao1024
6afbb33af1 fix(moa): add explicit warnings to reference prompt against claiming tool execution 2026-07-23 18:40:09 -07:00
sgtworkman
3dfe712384 fix(moa): scope quiet relay to machine-readable CLI
Keep MoA reference display events off the machine-readable -Q stdout
surface (platform=cli with tool_progress_mode=off) while preserving them
everywhere else. Extracts the relay into module-level helpers so the
policy is testable.

Salvaged from #67334.
2026-07-23 18:40:09 -07:00
oppenheimor
ca294d3e62 feat(moa): add reference model toggles 2026-07-23 18:11:57 -07:00
SquabbyZ
89e6f4c989 feat(agent): add MOA progress indicator (#59546)
Adds per-reference progress events and a phase-transition marker to the
MoA display pipeline so TUI / CLI / desktop surfaces can render a status
bar like `MOA: 2/3 refs done` and surface which phase (reference vs
aggregator) is currently active.

  - `moa.progress`  — fired once per reference completion with
                       `refs_done`, `refs_total`, and the source label
  - `moa.phase`     — fired on phase transitions (currently the single
                       `phase="aggregator"` transition once the fan-out
                       finishes)

Plumbed through the existing `reference_callback` →
`tool_progress_callback` → gateway path; no new UI surface. The legacy
`moa.reference` / `moa.aggregating` events are unchanged for backwards
compatibility.

AI-assisted fix by https://github.com/SquabbyZ/peaks-loop
2026-07-23 18:11:57 -07:00
hellofrommorgan
d6ffe3d767 fix(windows): hide console flashes from LSP server spawn and installer subprocesses
Salvaged from PR #47971 (LSP subset). On Windows, .cmd-wrapped language
servers (e.g. pyright-langserver.CMD launched via cmd.exe /c) and the
npm/go/pip LSP auto-installers spawn without CREATE_NO_WINDOW, so a
console window flashes whenever the spawn happens under a console-less
parent — e.g. a VS Code/Zed extension host running the ACP adapter.

- agent/lsp/client.py::_spawn: pass creationflags=windows_hide_flags()
  to the language-server asyncio subprocess (inert 0 on POSIX;
  start_new_session is kept — it is POSIX-only and ignored on Windows).
- agent/lsp/install.py: same flags on the npm and go installer
  subprocess.run calls. The pip path goes through
  hermes_cli.tools_config._pip_install, which already hides its windows.

Adapted from the PR's hand-rolled _NO_WINDOW constant to the repo's
hermes_cli._subprocess_compat.windows_hide_flags() convention.
2026-07-23 17:59:52 -07:00
Teknium
850f576f3d feat(moa): add every_n fanout cadence with cached-guidance reuse
Extends the fanout enum with 'every_n:<N>' (N >= 2): advisors run on the
first iteration of each user turn and every Nth tool iteration after it;
off-cadence iterations REUSE the cached guidance from the last on-cadence
run via the same cache mechanism the user_turn fanout uses, so the
aggregator still gets advice on every step. The cadence counter is scoped
per user turn (resets on a new user message) and only advances when the
advisory state actually changes, so streaming retries never consume a
cadence slot. Mapping form {mode: every_n, n: N} normalizes to the
canonical string. Unknown/degenerate values fall back to per_iteration.

Addresses issue #63393 (advisor fan-out multiplies turn latency/cost by
the tool-iteration count). Redesigned from PR #63448: the submitted shape
skipped references entirely on off-cadence iterations (aggregator ran
advice-less); this version keeps the last advice in play, credited for
the idea and cadence framing.

Config-gated, default-off (default fanout remains per_iteration).

Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
2026-07-23 17:50:40 -07:00