Commit graph

1298 commits

Author SHA1 Message Date
Ben Barclay
2ea39daeb1
fix(gateway): share relay adapter in multiplex mode (#65366) 2026-07-16 14:30:42 +10:00
Teknium
5f1991bf6b
fix(gateway): import PairingStore in _start_secondary_profile_adapters (#65118)
The served-profiles block in _start_secondary_profile_adapters references
PairingStore, but the class's only import in gateway/run.py is method-local
inside __init__ — so the reference raised NameError at runtime, silently
swallowed by the enclosing try/except ('could not record served_profiles').
Result: multiplexing gateways never created per-profile pairing stores, and
authz pairing checks for secondary profiles fell through to the global
whitelist. Also masked the served_profiles runtime-status write.

One-line fix (local import alongside write_runtime_status) + regression
tests that drive the real method and assert the stores materialize,
verified red without the import and green with it.

Surfaced during the profile-routing sweep by @CocaKova's PR #61689, which
included the same fix as part of a larger feature.
2026-07-15 12:11:16 -07:00
Jason
c346f018d4 fix(gateway): evict stale-self-heal agent cache entries pointing at dead sessions
The #54878 self-heal (SessionStore.get_or_create_session) drops a routing
key pointing at a session already ended in state.db and recovers/recreates
a fresh session_id under the same session_key. The #54947 fix (agent-cache
cache-hit guard in gateway/run.py) treats a cached agent whose snapshot
session_id differs from the current session_id, under the same
session_key, as an intentional /resume-/branch-style switch between two
live sibling conversations, and reuses it unchanged to protect the prompt
cache.

These two fixes compose incorrectly: when the #54878 self-heal just fired,
the cached agent's session_id is not a live sibling — it's the dead session
just routed away from. #54947's "different session_id -> reuse freely" rule
reuses it anyway. The stale agent runs the turn, and the post-run "session
split" sync (agent.session_id != session_id) then writes the routing key
straight back onto the dead session_id, undoing the self-heal. This repeats
on every subsequent message until an interrupt (e.g. /stop) happens to race
in before that post-run sync, silently discarding conversation context.

Reproduced live on the engineering gateway (2026-07-12, routing key
agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines
over ~40 minutes, each followed by the dead session_id being reused and
re-synced back, until an interrupted /stop finally let a fresh session
stick — at which point all prior context was gone.

No open upstream issue tracks this specific interaction as of 2026-07-12
(checked #54878, #54947, #59580, #59597, #61220 — all cover adjacent but
distinct edges of the self-heal / agent-cache system).

Fix: before applying #54947's reuse-on-mismatch rule, check (outside the
cache lock, via SessionStore._is_session_ended_in_db) whether the cached
snapshot's session_id is itself ended in state.db. If so, treat it as a
stale self-heal artifact and evict/rebuild fresh -- same as a genuine
cross-process write -- instead of reusing it. Re-validates the peeked
verdict against the tuple actually held under the lock so a race can't
apply a stale verdict to a different (possibly live) cache entry.

Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new
cases: dead-session eviction, live-sibling reuse preserved [#54947 intact],
cross-process invalidation preserved [#45966 intact], same-session_id dead
edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed,
9040 passed, 11 skipped -- all 14 failures verified pre-existing on
unpatched main (confirmed via git stash + re-run), unrelated to this
change.
2026-07-16 00:11:22 +05:30
Frowtek
170959d80a fix(gateway): sanitize sender-name prefix in shared multi-user sessions
GatewayRunner._prepare_inbound_message_text() interpolated
source.user_name — the platform-supplied, user-settable display name —
directly into the message text of every turn in a shared multi-user
session: f"[{source.user_name}] {message_text}". Shared sessions are the
default for any threaded conversation (thread_sessions_per_user defaults
to False) and apply to any group with group_sessions_per_user=False, so
no special configuration is needed to reach this path.

An unescaped display name containing embedded newlines could therefore
masquerade as a new markdown section (a fake "## Override" heading)
inside the live conversation the model reads on every turn — the same
indirect-prompt-injection vector gateway/session.py's
build_session_context_prompt() already guards against for the identical
user_name field via _format_untrusted_prompt_value(), which was never
applied to this sibling call site.

Add neutralize_untrusted_inline_text() alongside the existing helper in
gateway/session.py: it collapses embedded newlines/control characters to
a single inert line without JSON-quoting, so inline "[Name] message"
formatting is preserved byte-for-byte for the common case (unlike
reusing _format_untrusted_prompt_value directly, which would add visible
quote marks to every sender prefix). Wire it into the sender-prefix
construction in gateway/run.py.
2026-07-15 23:48:34 +05:30
Teknium
647520f83e fix(gateway): gate profile routing on multiplex_profiles + widen batch-key routing to all adapters
Follow-ups on the salvaged #20096 profile-routing feature:

- _profile_name_for_source now returns None unless gateway.multiplex_profiles
  is on. Routing stamps source.profile, which namespaces session/batch keys,
  but the profile-scoped agent run only activates under multiplexing — without
  the gate, configured routes with multiplexing off split batch/session keys
  into agent:<profile> while the agent still ran from agent:main.
- Widen the profile-aware _text_batch_key fix from Discord to every adapter
  that builds batch keys via build_session_key (telegram, whatsapp, matrix,
  feishu, wecom, weixin) — routing is platform-generic, so the batch-key
  namespace fix must be too.
- Downgrade the no-route-matched log from INFO to DEBUG (fired on every
  unrouted inbound message).
- GatewayConfig.to_dict(): serialize profile_routes as plain dicts
  (ProfileRoute dataclasses are not JSON-safe).
- Docs: correct the 'independent of multiplexing' claim in
  docs/profile-routing.md (routing requires multiplexing), fix the
  platform-only specificity row (0, not 1), and document profile_routes in
  website/docs/user-guide/multi-profile-gateways.md.
- Tests: pin the multiplex gate (routes ignored when off, active when on,
  build_source end-to-end stays in agent:main when off).
2026-07-15 09:50:05 -07:00
Burgunthy
f58e4622cf fix(gateway): profile routing — conjunctive matching + universal gateway_runner
Addresses hermes-sweeper review on #20096.

Problem 1 (profile_routing.py): route matching returned True on a chat_id
hit before the guild_id constraint was consulted, so a route declaring both
guild_id and chat_id matched on chat_id alone. Restored conjunctive (AND)
semantics — every declared discriminator must hold; hierarchical parent_chat_id
matching is preserved. Added a regression test for the guild+chat case.

Problem 2 (base.py / run.py): gateway_runner was injected only when an adapter
pre-declared the attribute, and only Discord did — so build_source never called
_profile_name_for_source for Telegram/Feishu/Slack/etc., despite the
platform-generic claim. Declared gateway_runner on BasePlatformAdapter and made
the plugin-registry injection unconditional, so profile routing now reaches
every platform. Added non-Discord (Telegram) resolution coverage and an
injection-inheritance test.

Also adds docs/profile-routing.md documenting gateway.profile_routes
(matching rules, specificity, profile isolation) — requested in review.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-15 09:50:05 -07:00
Burgunthy
a1d6654264 fix(gateway): read adapter token from config for fingerprint check
_adapter_credential_fingerprint only looked at adapter.token directly,
but Discord (and similar) adapters store the bot token on their config
sub-object, not on self. Every Discord adapter in a multiplexed
gateway therefore returned None, the same-token conflict check was
silently skipped, and N adapters all polled the same bot token —
producing a per-message race where whichever adapter won the GIL
answered the user.

Adds a config-token fallback (token, then bot_token) so the check
actually fires for config-backed adapters. Direct adapter.token
still takes precedence when both exist.

Tests cover: config-backed token produces a fingerprint, distinct
tokens produce distinct fingerprints, direct token wins over config,
config without token attributes returns None.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-15 09:50:05 -07:00
Burgunthy
9166d727b8 fix(profile-routing): remove dead forum cache, warn on missing profile
Three follow-ups to the initial routing PR after code review:

1. Remove dead forum-post hierarchy cache. The `_forum_post_cache`,
   `register_forum_post()`, and `resolve_forum_channel()` were never
   wired up — no caller in the codebase. Discord's adapter already
   sets `parent_chat_id` to the immediate parent (forum channel for a
   forum post), so the existing `self.chat_id == parent_chat_id`
   branch in `matches()` handles forum posts correctly without a
   cache. The hierarchical-resolution branch in `matches()` and the
   bounded-LRU infrastructure are removed.

2. Fix docstring specificity numbers (8 → 14, 4 → 6) and rewrite the
   "Hierarchical matching" section to describe the actual one-level
   parent_chat_id behavior. Removed unused `OrderedDict` and `Set`
   imports.

3. Warn loudly when a routed profile doesn't exist on disk. Previously,
   a typo in `profile_routes` (e.g. `crypto-tradr`) silently fell back
   to the global HERMES_HOME, causing the message to read the default
   profile's memory/credentials with no signal to the operator. Now
   emits a `logger.warning` with the profile name, source identifier,
   and the fallback reason. Bare-exception path also gets `exc_info`.

Tests:
- test_profile_routing.py: +2 tests verifying forum post matching via
  direct parent_chat_id (covers the case the removed cache was meant
  for). 31 total, all pass.
- test_profile_resolution.py: NEW, 12 tests covering resolution order
  (source.profile > routing > active > default), missing-profile
  warning, exception handling, and routing consultation. All pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-15 09:50:05 -07:00
Burgunthy
5e65f6d79f feat(gateway): add profile-based routing for inbound messages
Adds gateway.profile_routes config that routes specific Discord
guilds/channels/threads (and other platforms) to different profiles.
The routing engine uses hierarchical specificity matching
(thread > channel > guild) with bounded LRU caching for forum post
resolution.

Routing result is stamped on source.profile by BasePlatformAdapter
.build_source() at inbound time. When gateway.multiplex_profiles is on,
the existing _profile_runtime_scope machinery picks up source.profile
and runs the whole turn inside the profile's HERMES_HOME — so memory,
skills, config, and secrets all resolve to that profile automatically.
No new isolation code is added; this PR only adds the routing decision
layer on top of the existing multiplexing infrastructure.

Configuration:

    gateway:
      multiplex_profiles: true
      profile_routes:
        - name: server-default
          platform: discord
          guild_id: "GUILD_ID"
          profile: server-profile
        - name: special-channel
          platform: discord
          guild_id: "GUILD_ID"
          chat_id: "CHANNEL_ID"
          profile: channel-profile

When multiplex_profiles is off, profile_routes is ignored (no behavior
change for single-profile gateways).

Tests: 29 unit tests covering specificity scoring, hierarchical
matching, path-traversal validation, and config parsing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-15 09:50:05 -07:00
Teknium
09b6d22dfe fix(gateway): harden hidden-incomplete detection against the sentinel final_response
Follow-up to the salvaged #51657: the conversation loop returns the
retry-exhaustion sentinel as BOTH final_response and error, so the
original detector (which required final_response to be falsy) never
fired on real exhaustion turns — the sentinel text was delivered
verbatim into the channel, exactly the #51628 poisoning vector. Detect
the sentinel echo, blank it before empty-response normalization, and
never suppress a turn whose final_response is genuine model text.
Also: dedupe-guard mock fix in the test fixture (has_platform_message_id
must return False, not a truthy MagicMock) and two guard tests
(real answer never suppressed; interrupted/failed never classified).
2026-07-15 09:48:24 -07:00
LeonSGP43
08015c3a8f fix(gateway): suppress hidden-only incomplete codex turns 2026-07-15 09:48:24 -07:00
KCAYAAI
e9d564c09c fix(gateway): resume typing after clarify reply 2026-07-14 21:31:32 -07:00
kshitijk4poor
4554fe128a fix(slack): complete agent view workspace routing 2026-07-14 13:58:36 -07:00
Teknium
e81d18dfb4 refactor(reasoning): unify per-model reasoning resolution behind a single chokepoint
Collapse the six per-surface copies of override-then-global resolution
(CLI startup, gateway, TUI, cron, /model switch, fallback activation)
onto one shared resolve_reasoning_config() in hermes_constants.

Also fixes the gateway resolving reasoning against config model.default
instead of the session's effective model: after a session-only /model
switch, the switched model's override now applies (gateway message paths
pass the resolved session model through _resolve_session_reasoning_config;
/reasoning status reads the session model override).

Cleanup: drop docs/PER_MODEL_REASONING.md (duplicates the website docs
page), drop the change-detector _config_version test (no bump needed —
deep-merge handles new keys), remove a stale plan-reference comment.

Adds chokepoint contract tests (13) and gateway session-effective-model
regression tests (2).
2026-07-14 11:46:40 -07:00
ScotterMonk
d9cdb81923 feat(config): support per-model reasoning_effort overrides
Add agent.reasoning_overrides dict to config.yaml. Users can now set
a reasoning_effort per model, overriding the global agent.reasoning_effort.

Example:
  agent:
    reasoning_effort: "medium"       # global default
    reasoning_overrides:
      "openrouter/anthropic/claude-opus-4.5": "xhigh"
      "openai/gpt-5": "low"
      "claude-sonnet-4.6": "high"    # bare model name also works

The helper is spelling-tolerant: override keys match regardless of
provider prefix or dots-vs-dashes normalization, so users can write
keys in any sensible form and they'll match.

Resolution priority:
1. Session-scoped /reasoning --session override (gateway only; unchanged)
2. Per-model override from agent.reasoning_overrides (spelling-tolerant)
3. Global agent.reasoning_effort (existing)
4. Provider default (unchanged)

Wired into:
- CLI startup (cli.py)
- Messaging gateway agent construction (gateway/run.py)
- Desktop/TUI _load_reasoning_config (tui_gateway/server.py)
- Cron job scheduler (cron/scheduler.py)
- /model mid-session switch (agent/agent_runtime_helpers.py)
  + _primary_runtime now tracks reasoning_config for correct fallback recovery
- Fallback activation (agent/chat_completion_helpers.py::try_activate_fallback)
  + Re-resolves reasoning_config for the fallback model (best-effort)

Closes #21256 (per-model reasoning_effort defaults).

Note: no hermes config set agent.reasoning_overrides.<model> support;
users edit the YAML directly. _set_nested splits on "." and would
corrupt model keys containing version dots.
2026-07-14 11:46:40 -07:00
dsad
fd461b58ca fix(gateway): fail closed on compression state probe errors 2026-07-14 16:56:34 +05:30
kshitijk4poor
0b422559f3 fix(session): preserve clean multimodal persistence override 2026-07-14 03:32:45 +05:30
kshitijk4poor
ffc10cc659 fix(gateway): quiesce API and cron work during drains
Reserve API requests before their first await, include every drain-owned work type in runtime status, and pause local cron dispatch while a gateway drain is active.
2026-07-13 23:26:10 +05:30
kshitijk4poor
104ffeae23 fix(gateway): complete API-server shutdown drain 2026-07-13 23:26:10 +05:30
Bartok9
021ee34546 fix(gateway): drain in-flight api_server runs on shutdown
Closes #63529

Root cause: GatewayRunner._drain_active_agents only waited on
_running_agents + cron in-flight counts. Desktop/API sessions are
tracked solely inside APIServerAdapter (_inflight_agent_runs +
_active_run_agents), so stop/restart logged active_at_start=0 and
systemd SIGKILL'd mid-tool work.

Fix: APIServerAdapter.active_agent_work_count() plus
GatewayRunner._active_api_run_count() folded into the drain wait,
status updates, timeout result, and shutdown logs — same pattern as
_active_cron_job_count for #60432.

Verification: pytest tests/gateway/test_api_server_active_work_drain.py
tests/gateway/test_cron_active_work_drain.py -q → 19 passed
2026-07-13 23:26:10 +05:30
teknium1
d0e9a42cec fix(delegation): harden durable completion delivery 2026-07-13 07:28:21 -07:00
teknium1
67f4e1b4a9 feat(delegation): persist background completions 2026-07-13 07:28:21 -07:00
John Lussier
94a7705bdd fix(gateway): deduplicate completion delivery 2026-07-13 07:28:21 -07:00
Teknium
d48bf743f2 fix(approval): scope smart deny owner overrides to one operation
Co-authored-by: Sergei Ivanov <kavi@local.hermes>
2026-07-13 04:31:55 -07:00
Teknium
7550c594ce
feat(reasoning): add max and ultra effort levels (#62650) 2026-07-12 00:26:49 -07:00
kshitijk4poor
4281151ae8 test(gateway): cover effective context budget 2026-07-11 23:46:13 +05:30
kshitijk4poor
265ac7d812 fix(gateway): honor runtime context budgets 2026-07-11 23:46:13 +05:30
kshitijk4poor
ff84e3f7d5 fix(gateway): scope queued context references 2026-07-11 23:46:13 +05:30
kshitijk4poor
37a942650f fix(gateway): scope context refs to runtime profile 2026-07-11 23:46:13 +05:30
TheTom
4df6e6280d fix(gateway): @ context reference expansion never ran (AttributeError)
GatewayRunner._prepare_inbound_message_text's "@" context-reference
block read self._model / self._base_url to resolve the model for
get_model_context_length_async. GatewayRunner never sets either
attribute (copy-pasted from HermesCLI in da44c196b, which does carry
self.model/self.base_url). Every message containing "@" raised
AttributeError inside the try block, silently swallowed by the
surrounding except Exception at debug level, so
preprocess_context_references_async never ran and @file:/@folder:/@diff/
etc. references passed through to the model unexpanded.

Fix: resolve model/provider/base_url via
self._resolve_session_agent_runtime(source=, session_key=,
user_config=), the same session-aware resolution the hygiene
compression block already uses a few hundred lines later in this file.

Also raise the swallow log from debug to warning (with exc_info at
debug) so a future regression here is visible instead of silent.
2026-07-11 23:46:13 +05:30
kshitijk4poor
9d38a2309e fix(gateway): enforce one async SessionStore boundary 2026-07-10 12:38:48 +05:30
kenyonxu
94c2a4016b fix(gateway): offload both blocking sources in compression-in-flight check (#5)
The sync _session_has_compression_in_flight sat on the message hot path
and blocked the event loop twice: under session_store._lock during
_ensure_loaded_locked (JSON read) and via db.get_compression_lock_holder
(SQLite SELECT). Async-ify the method and offload both sources via
asyncio.to_thread; await the call site in _handle_active_session_busy_message.
2026-07-10 12:38:48 +05:30
kenyonxu
24ea21993f fix(gateway): offload session store calls off the event loop via asyncio.to_thread
Every inbound message calls get_or_create_session which synchronously
executes _is_session_ended_in_db → db.get_session → conn.execute on
the asyncio event loop. On a ~1.4GB state.db, this blocks the loop
for seconds to minutes, starving Discord heartbeats.

Upstream #55159 fixed the same pattern for self._session_db in
gateway/run.py but missed SessionStore._db in gateway/session.py.

This follows the exact same approach as #55159:
- session.py internals stay fully synchronous (zero changes)
- Threading.Lock contract is preserved
- All hot-path callers in run.py and slash_commands.py wrap calls
  with await asyncio.to_thread(self.session_store.method, ...)

Affected: get_or_create_session, switch_session, update_session,
load_transcript, rewrite_transcript, rewind_session, reset_session,
set_model_override, _save — ~24 call sites across 2 files.

# Conflicts:
#	gateway/slash_commands.py
2026-07-10 12:38:48 +05:30
teknium1
a0972b9748 fix: widen None-deref guards to config-derived sibling sites + tests
Sibling sites of the salvaged #55997 fix, all reading user-editable
config values through .get(key, '').method(): MoA slot provider/model
labels, gateway quick-command alias targets (2 sites), gateway.proxy_url,
and gateway.relay_url. Regression tests for the contributor's two sites
plus the MoA labels.
2026-07-09 21:10:07 -07:00
AlexFucuson9
549b87c9aa fix(gateway): prevent hygiene compression from destroying archived transcript
When gateway session-hygiene auto-compression fires with in-place
compaction, the flow was:

1. _compress_context() calls archive_and_compact() — soft-archives old
   rows (active=0, compacted=1) and inserts compacted messages as the
   new active set.  This is the non-destructive, durable path.

2. The hygiene handler then called rewrite_transcript() — which calls
   replace_messages(active_only=False) — DELETEing ALL rows including
   the just-archived turns.  Silent permanent data loss (#61145).

The interactive /compress handler had the same bug.

Fix: only call rewrite_transcript() when session rotation produced a new
session id (legacy path).  When in-place compaction succeeded, skip the
rewrite — archive_and_compact() already handled persistence.

Closes #61145.
2026-07-09 18:04:00 -07:00
kshitijk4poor
0a01b2087d fix(gateway): harden fallback-chain refresh from review findings
Follow-ups on the #60987 salvage (review pass):
- _refresh_fallback_model: keep last known-good chain on transient
  config.yaml read/parse failure (user mid-edit, torn write) — only a
  successful read that lacks the key clears the chain. Previously a
  refresh error wiped a cached agent's working fallback for the turn.
- Move the cached-agent refresh+apply OUTSIDE the agent-cache lock:
  config.yaml read is disk I/O and the idle-sweep watcher contends on
  that lock (same reasoning as #52197). Per-session turn serialization
  keeps the post-lock apply safe.
- _apply_fallback_chain_to_agent: clear _unavailable_fallback_keys when
  chain content actually changes, so an entry re-configured mid-uptime
  (e.g. credentials added) is retried instead of staying suppressed for
  the cached agent's lifetime; no-op refreshes keep the memo.
- Tests: cwd-independent source pin (Path(__file__) anchor), pin the
  reuse-path apply call, + regression tests for last-known-good, memo
  clear-on-change, memo keep-on-unchanged (mutation-verified).
2026-07-09 02:22:12 +05:30
HexLab98
be1346cf2a fix(gateway): reload fallback_providers on live agent create/reuse
Gateway froze the fallback chain at process start while cron reloads it
per job, so a chain configured after hermes gateway was running never
reached messaging sessions. Refresh from disk on agent create and when
reusing a cached agent.

Fixes #60955.

(cherry picked from commit b64e7155b2)
2026-07-09 02:22:12 +05:30
nullptr0807
d6a275b735 fix(gateway): compact hygiene transcripts in place 2026-07-08 12:35:50 -07:00
teknium1
75efd73961 fix(gateway): never resurrect ended sessions for delegation completions; /new severs in-flight delegations
Completes the session-binding class on the gateway surface (#55578),
matching the TUI rules:

1. Fail-closed pinning: switch_session() re-opens ended sessions, so
   pinning a completion to a spawning session that has since ENDED
   (user /new, closed rotation) would resurrect a conversation the user
   explicitly ended and inject into it. The injection path now checks
   the pinned row's ended_at first and drops the injection with a
   WARNING when the spawning session is dead or unknown - the result
   stays in the delegation records.

2. /new ends the old conversation's delegations: _handle_reset_command
   calls interrupt_for_session() with the expiring durable session id
   (matching the parent_session_id pin stamped at dispatch) plus the
   routing key as fallback, so a reset can't leave dangling subagents
   whose completions have no live owner.

interrupt_for_session() gains the parent_session_id selector because a
gateway chat's session_key (the platform conversation key) survives a
reset while the session id rotates - key-based matching alone could
never sever a gateway conversation's delegations.
2026-07-08 08:10:28 -07:00
nankingjing
d39c62409b fix(delegate): pin async completion to spawning parent session (#57498)
Background delegate_task completions only carried session_key. When multiple
active sessions shared a routing peer, get_or_create_session could recover the
latest ended_at IS NULL row and inject the subagent result into the wrong
session.

Capture parent_agent.session_id at dispatch time, include it on async-delegation
completion events, and pin gateway routing via switch_session when the
synthetic completion message is handled.

Fixes #57498
2026-07-08 08:10:28 -07:00
Tony Simons
f75f3cd713 fix(delegation): route async delegate_task results back to originating session
The completion event already carries the dispatching session's session_key
(captured at dispatch time in delegate_tool.py:2798), but the delivery
router ignored it — results landed in whatever session was active at
completion time instead of the session that dispatched the subagent.

Changes:
- drain_notifications() in process_registry.py: optional session_key
  filter. Non-matching async_delegation events are re-queued instead of
  consumed, so they remain available for the correct session's drain.
- cli.py process_loop: passes active session_key to drain_notifications()
- tui_gateway/server.py post-turn drain: passes session_key from the
  TUI session dict
- gateway/run.py _build_process_event_source: logs warning when routing
  metadata is unresolvable (previously silent drop)
- Regression tests verifying session-scoped drain filtering

Fixes #58684
2026-07-08 07:39:52 -07:00
joaomarcos
24e9ed73c2 fix(gateway,cron): make shutdown drain visible to in-flight cron work
Cron jobs run through cron/scheduler.py's own ThreadPoolExecutor via a
standalone AIAgent (run_job/run_one_job), entirely outside
GatewayRunner._running_agents -- the dict _drain_active_agents() and
every other active-work check on that class reads. A gateway shutdown
(/update, /restart, and SIGUSR1 all funnel through the same stop())
could log active_at_start=0 and immediately kill tool subprocesses
while a cron job's terminal command was still running, with no wait
and no indication anything was interrupted.

Real-world impact (from the issue): a scheduled daily briefing cron
job was in flight during /update, its tool subprocess got killed
by the unconditional shutdown cleanup, and the job was never marked
failed -- it simply never completed or delivered, with no error
surfaced anywhere. A repro with a 30-minute `sleep` cron job in flight
during /update reproduced the same pattern: subprocess killed at
+0.22s of drain (active_at_start=0), the job's agent thread continued
in-process and produced a plausible-looking final response from the
truncated tool output, and the scheduler marked the run successful.

Root cause is layered, not a single line:

1. GatewayRunner._drain_active_agents() only waits on _running_agents.
   Cron work was invisible to it, so drain returned instantly whenever
   the only active work was a cron job.
2. Even with visibility, the shutdown's final tool-subprocess kill
   (process_registry.kill_all()) is a global, unconditional sweep with
   no per-job targeting -- a long-running cron job that outlives the
   drain timeout still gets its subprocess killed.
3. cron/scheduler.py had no way to detect that a job's tool subprocess
   was killed out from under it mid-run; the agent thread kept going
   and its eventual (often degraded but plausible-looking) response
   got reported as a normal successful completion.

Fix, three parts:

- cron/scheduler.py: expose get_running_job_ids() (thread-safe
  snapshot of the existing _running_job_ids set, already used to
  prevent double-dispatch) so the gateway can read cron's in-flight
  state without reaching into private module internals.

- gateway/run.py: GatewayRunner._active_cron_job_count() reads that
  snapshot. _drain_active_agents() now waits on
  (_running_agents OR active cron jobs), so a cron-only workload gets
  the same bounded wait chat sessions already get instead of an
  instant active_at_start=0. Shutdown drain logging gains
  cron_active_at_start/cron_active_now fields alongside the existing
  ones (unchanged, for compat).

- cron/scheduler.py: mark_running_jobs_interrupted(reason), called by
  gateway/run.py's _kill_tool_subprocesses() right after
  process_registry.kill_all(), marks every job still in
  _running_job_ids at that instant as failed/interrupted via the
  existing mark_job_run() -- and records the job IDs in
  _interrupted_job_ids BEFORE writing, so run_one_job()'s own
  eventual completion for the same run (racing in its own thread)
  checks that flag and skips its normal write instead of clobbering
  the interrupted status with a false "ok" produced from the
  now-truncated tool output. This does not attempt to correlate a
  killed PID to a specific job ID (process_registry tracks PIDs, not
  job IDs) -- any job still dispatched at the moment of a forced kill
  is treated as interrupted, matching the existing coarser precedent
  set by _interrupt_running_agents(), which interrupts every entry in
  _running_agents on a drain timeout without per-agent correlation
  either.

Deliberately out of scope (flagged in the issue as a separate,
lower-priority concern): startup-time reconciliation of cron runs that
started but never reached a terminal status.

Testing:

- tests/cron/test_shutdown_interrupt.py (12 tests): get_running_job_ids
  snapshot semantics, mark_running_jobs_interrupted marking/no-op/
  partial-failure behavior, and -- the core race guard -- run_one_job
  skipping its own last_status write (both the success path and the
  exception path) when the shutdown path already marked the run
  interrupted, with a control test proving ordinary un-interrupted
  completions are unaffected.

- tests/gateway/test_cron_active_work_drain.py (9 tests):
  _active_cron_job_count reading cron state and failing closed (0) if
  the cron module is unavailable; _drain_active_agents waiting for an
  in-flight cron job the same way it waits for chat sessions, timing
  out if the job outruns the window, and leaving existing chat-session
  drain behavior unchanged; a full runner.stop() integration test
  (drain-timeout path) proving mark_running_jobs_interrupted actually
  fires with the right job ID when a tool subprocess is force-killed,
  plus a no-op control when nothing cron-related is in flight.

- tests/gateway/test_shutdown_cache_cleanup.py: added
  _active_cron_job_count() to that file's hand-rolled _FakeGateway test
  double, which stop() now calls -- without it those 8 pre-existing
  tests AttributeError (caught by fail-then-pass below, not a
  production bug).

Fail-then-pass: reverted gateway/run.py + cron/scheduler.py, all 21
new tests fail (fixture/attribute errors -- the feature doesn't exist
yet); restored, all 21 pass.

Regression check: ran the full plausibly-affected surface --
tests/gateway/{test_gateway_shutdown,test_restart_drain,
test_restart_notification,test_restart_redelivery_dedup,
test_restart_resume_pending,test_restart_service_detection,
test_shutdown_cache_cleanup,test_stuck_loop,test_clean_shutdown_marker,
test_external_drain_control,test_session_state_cleanup,
test_update_command,test_update_streaming}.py plus tests/cron/ (944
tests) -- against a clean upstream/main checkout and against this
branch. Diffed the two FAILED lists: identical, 20 pre-existing
failures on both sides (Windows-locale/cp1252 file-encoding issues and
Unix-permission-bit assertions that don't apply on this Windows dev
box), zero new failures, zero fixed-by-accident. The 8
test_shutdown_cache_cleanup.py failures found mid-development were
from the _FakeGateway gap above, fixed in the same commit and
confirmed clean on the final rerun (diff against baseline: exit 0).

Fixes #60432
2026-07-07 22:15:04 -07:00
HexLab98
862aee4956 fix(gateway): drain in-flight cron jobs before shutdown tool kill
/update and other shutdown paths only waited on gateway session agents,
so active cron tool work was killed immediately in final-cleanup while
the scheduler could still mark the job successful (#60432).
2026-07-07 22:15:04 -07:00
kshitijk4poor
4ca61869cb docs: sharpen bypass comment per review
The live path bypasses the whole _build_gateway_agent_history cleanup
pipeline, not just the replay-cleanup pass — say so precisely.
2026-07-08 00:59:56 +05:30
pierrenode
d33becd877 fix(gateway): demote PRIORITY-path interrupt to queue during compression (#56391)
_handle_active_session_busy_message (the busy_session_handler most
platform adapters register) demotes busy_input_mode='interrupt' to
queue semantics for two reasons: active subagents (#30170) and, as of
this week, context compression in flight (#56391) — interrupting while
compression holds the state.db lock races a new turn against the
pre-rotation parent session, and if that new turn also grows past the
compression threshold it starts its own uncancellable compression on
the same stale parent, forking orphaned compression siblings.

_handle_message has its own, independent inline "PRIORITY" busy-path
(reached directly with a live running agent — see the `if _quick_key in
self._running_agents:` guard, exercised end-to-end by the existing
tests/gateway/test_running_agent_session_toggles.py harness). Its own
comment says it mirrors _handle_active_session_busy_message's subagent-
demotion rationale verbatim, and it does demote for active subagents,
but it never checked _session_has_compression_in_flight, so a plain-text
follow-up landing on this path while compression is mid-flight still
called running_agent.interrupt() unconditionally.

Fix: add the same _session_has_compression_in_flight(session_key) check
before the PRIORITY interrupt call, demoting to queue exactly like the
sibling path.

Tests: tests/gateway/test_priority_path_compression_demotion_56391.py
drives _handle_message end-to-end (reusing the test_running_agent_
session_toggles.py harness pattern) with a live running agent and a
mocked compression lock. Mutation-verified: reverting the fix makes the
demotion test fail (interrupt() gets called) against the pre-fix code;
a control test pins the unchanged default-interrupt behavior when no
compression lock is held.
2026-07-07 21:06:07 +05:30
teknium1
1deeaf71ab fix(discord): truncate thread titles by UTF-16 units + AUTHOR_MAP
Discord thread names share the same UTF-16 component budget as select
labels and buttons — route the sanitizers in gateway/run.py and the
adapter's rename_thread through utf16_len/_prefix_within_utf16_limit
instead of code-point slices. Adds rungmc357 to AUTHOR_MAP.
2026-07-07 05:11:59 -07:00
Georgio Constantinou
0d9ed9214d Add semantic titles for Discord auto-threads 2026-07-07 05:11:59 -07:00
kshitij
7647eff360
Merge pull request #60117 from kshitijk4poor/fix/59607-cached-agent-expiry
fix(gateway): re-apply confirmation expiry on the cached-agent live-history path (#59607)
2026-07-07 15:14:24 +05:30
teknium1
87b65e24a7 refactor(compression): scope Codex-native compaction to the app-server runtime
Drop the Responses-API native compaction path and its opt-in umbrella
flag from the salvaged feature. On the Codex OAuth chat route Hermes
owns the message list and the summary compressor works (and stays
provider-portable — encrypted compaction items would lock the session
history to chatgpt.com and break /model switches and provider
fallback). On the app-server runtime (codex CLI/agent) the codex agent
owns the real thread context, so thread/compact/start is the only
mechanism that can actually shrink it (#36801) — that path is now the
default behavior for codex_app_server sessions, controlled by
compression.codex_app_server_auto (native|hermes|off), no umbrella
flag.

Removed: responses.compact() call path, codex_compaction_items replay/
persistence plumbing, codex_native_compaction + codex_responses_threshold
config keys, desktop settings fields, and their tests. Kept: everything
app-server (compact_thread(), compaction notifications, bookkeeping,
docs, tests) plus cache-busting keys for the surviving knobs.
2026-07-07 02:39:54 -07:00
kshitijk4poor
3c8130a826 fix: re-apply confirmation expiry on the cached-agent live-history path
Review finding: when the FTS write-corruption guard (#50502) prefers the
cached agent's live _session_messages over the reloaded transcript, that
history bypasses the replay-cleanup pass in _build_gateway_agent_history
— a stale dangerous confirmation could slip through unredacted on the
same-process salvage path. Re-apply the (idempotent) expiry stripper to
the selected live history.
2026-07-07 14:46:10 +05:30