The disease: ~15 scattered raw yaml.safe_load(config.yaml) reads that
silently miss managed-scope overlay, ${ENV_VAR} expansion, profile-aware
pathing, and root-model normalization. Every new config feature needed an
N-site sweep (incident chain 9cbcc0c9c8 → 732293cf87 → b0e47a98f9 →
1928aa0443). This commit assigns every raw read to an owner and adds a
lint-guard test so the class cannot regrow.
New primitive (additive-only change to hermes_cli/config.py):
read_user_config_raw(path=None) — reads the user file EXACTLY as
written; docstring states it is ONLY legal for write-back round-trips
and raw-file diagnostics. Behavioral reads must use
load_config()/load_config_readonly().
BEHAVIOR FIXES (class-a sites migrated to a canonical loader — these
previously read values that could DIFFER from the effective config):
gateway/run.py _try_resolve_fallback_provider → _load_gateway_runtime_config
keys: fallback_providers/fallback_model (provider, model, base_url,
api_key). Drift fixed: a managed-pinned fallback chain was ignored;
an api_key of "${OPENROUTER_API_KEY}" reached the resolver unexpanded.
gateway/run.py GatewayRunner._load_provider_routing → same loader
key: provider_routing. Drift fixed: managed-pinned routing prefs and
${VAR} templates were ignored.
gateway/run.py GatewayRunner._load_fallback_model → same loader
keys: fallback chain. Same drift as above.
gateway/run.py GatewayRunner._refresh_fallback_model
keeps the raw primitive (its last-known-good-on-parse-failure contract
forbids the fail-open loader, which returns {} on a torn write) but now
applies managed overlay + env expansion inline. Drift fixed: chain
edits under managed scope / env templates were previously frozen out.
tui_gateway/server.py _load_cfg (72 behavioral call sites)
now = raw read + managed overlay (pre-existing) + NEW ${VAR} expansion,
split from a new _load_cfg_raw() write-back primitive. Drift fixed:
e.g. custom_prompt: "hello ${VAR}", agent.system_prompt, model,
api_key/base_url templates reached sessions unexpanded. DEFAULT_CONFIG
is deliberately NOT merged (callers treat missing keys as unset;
`_load_cfg() == {}` sentinels and _save_cfg round-trips depend on it).
tui_gateway/server.py _profile_configured_cwd
keys: terminal.cwd of a NON-launch profile. Drift fixed: managed
overlay + ${VAR} expansion now apply (load_config() would resolve the
wrong profile's home, so the raw primitive + inline pipeline is used).
plugins/platforms/telegram/adapter.py _reload_dm_topics_from_config
→ load_config_readonly(). keys: platforms.telegram.extra.dm_topics.
Drift fixed: managed overlay + profile-aware pathing + expansion.
plugins/memory/holographic _load_plugin_config → load_config_readonly().
keys: plugins.hermes-memory-store.*. Same drift class.
WRITE-BACK ROUND-TRIPS (class-b: stay raw BY DESIGN via read_user_config_raw;
merging defaults/overlay would pollute the saved user file):
gateway/slash_commands.py: model persist x2, _save_gateway_config_key,
memory/skills write_approval toggles
gateway/platforms/yuanbao.py auto-sethome
tui_gateway/server.py _write_config_key + all cfg→_save_cfg blocks
(reasoning show/hide/full/clamp, details_mode[.section], prompt)
→ new _load_cfg_raw()
plugins/memory/holographic save_config
RAW-FILE DIAGNOSTICS + presence-sensitive bridges (class-c: stay raw,
now via the shared primitive with an explanatory comment):
hermes_cli/doctor.py x5 (model validation, stale-root-keys, .env drift,
deprecation sweep, memory-provider probe — the latter two keep their
inline managed overlay where they had one)
gateway/run.py _bridge_max_turns_from_config and the module-level
TERMINAL_*/HERMES_* env bridge (bridging merged defaults would export
all of DEFAULT_CONFIG into the environment; both keep their inline
overlay + expansion)
hermes_cli/send_cmd.py env bridge (same presence-sensitivity)
hermes_cli/gateway.py multiplex-conflict probe (reads the DEFAULT root's
config, not the active profile's — load_config is the wrong owner)
hermes_cli/profiles.py / hermes_cli/web_server.py / tools/wake_word.py
multi-profile reads (load_config targets only the ACTIVE profile home)
cron/jobs.py _resolve_default_model_snapshot and cron/scheduler.py
run_job config read keep their existing inline overlay+expansion but
now share the primitive (their fail-open + last-value semantics and
the deliberate no-defaults merge are preserved exactly).
Failure-semantics audit: every migrated site preserves its exact previous
behavior on missing file ({} / early return) and parse failure (raise into
the caller's existing except, warn, last-known-good, or fail-open) —
read_user_config_raw intentionally mirrors bare open()+safe_load semantics
(raises on parse errors, {} only on FileNotFoundError/non-dict root).
Guard: tests/hermes_cli/test_config_read_guard.py scans the tree for
yaml.safe_load within 6 lines of a 'config.yaml' reference outside an
explicit ALLOWLIST (hermes_cli/config.py, gateway/config.py, gateway/run.py
fallback path, hermes_cli/managed_scope.py which reads the MANAGED file,
gateway/readiness.py parse-health probe) and fails on new offenders.
E2E: tests/hermes_cli/test_config_loader_e2e.py runs a subprocess with a
temp HERMES_HOME (config.yaml containing ${E2E_PROMPT_SUFFIX}) plus a
HERMES_MANAGED_DIR overlay pinning agent.reasoning_effort, asserting
tui _load_cfg resolves "hello world"/"high" while _load_cfg_raw +
_save_cfg round-trip the template and user value verbatim with no
managed/default leakage.
Same class of bug as the LINE adapter (NS-603): defaulting the webhook
bind to "0.0.0.0" (or hardcoding it) binds IPv4 ONLY, so the listener
is unreachable over IPv6-only private networks such as Fly.io 6PN.
- wecom callback_adapter: DEFAULT_HOST None; config.py env seed no
longer forces 0.0.0.0 when WECOM_CALLBACK_HOST is unset.
- msgraph_webhook: DEFAULT_HOST None; the allowed_source_cidrs
requirement still fires for the all-interfaces default (host=None is
treated as network-accessible).
- whatsapp_cloud: DEFAULT_WEBHOOK_HOST None.
- teams: hardcoded 0.0.0.0 TCPSite bind → _DEFAULT_HOST=None with new
TEAMS_HOST / extra.host override (mirrors LINE_HOST pattern).
- telegram: hardcoded listen="0.0.0.0" → default "" (tornado
bind_sockets opens one socket per address family; verified against
PTB 22.6/tornado) with new TELEGRAM_WEBHOOK_HOST / extra.webhook_host
override.
Explicit host overrides everywhere are preserved; empty/unset collapses
to the dual-stack default. "::" remains a bad substitute on
bindv6only=1 hosts (see LINE adapter comment).
Add an opt-in streaming-audio adapter seam to BasePlatformAdapter so
voice-capable gateway platforms (LiveKit, Discord voice, future adapters)
can consume LLM output as streaming PCM audio before the full response
completes, dropping perceived voice latency from ~2-3.5s to ~500-800ms.
Adapter contract (gateway/platforms/base.py):
- AudioFormat dataclass: declared sample_rate, channels, sample_width
- StreamingTTSHandle: opaque handle with audible/aborted flags
- supports_streaming_tts / begin_streaming_tts / write_streaming_tts
/ finish_streaming_tts / abort_streaming_tts
- All default to unsupported/no-op so existing adapters are source-compatible
- Per-turn _streaming_tts_completed_chats set suppresses duplicate whole-file
auto-TTS when streaming succeeded; cleared after turn completion
Gateway consumer (gateway/streaming_tts_consumer.py):
- StreamingTTSConsumer: bridges sync agent deltas to async adapter audio sink
- Uses existing SentenceChunker (no competing parser)
- Thread-safe bounded queue; on_delta never blocks the agent worker thread
- Resolves configured streaming provider via resolve_streaming_provider()
- Serialises clause playback in order; flushes tail on completion
- Pre-audio failure: completed=False (falls back to whole-file TTS)
- Post-audio failure: completed=True, partial=True (no replay from start)
- Abort is idempotent; late chunks silently dropped
- Per-turn state isolated across concurrent chats
Gateway integration (gateway/run.py):
- message_type parameter threaded through _run_agent -> _run_agent_inner
- StreamingTTSConsumer created when voice input + auto-TTS + provider active
- Delta callback teed to both text stream consumer and TTS consumer
- TTS-only delta callback installed when text streaming is off
- finish() called from executor; wait_complete() in async context after
- Barge-in aborts the consumer at all three interrupt detection points
- Runner-level _send_voice_reply suppressed when streaming TTS completed
Tests (tests/gateway/test_streaming_tts_consumer.py):
- 15 focused tests: adapter defaults, lifecycle, ordered chunks,
unsupported/No-streamer fallback, abort idempotency, late-chunk drop,
pre/post-audio failure, concurrent-turn isolation, think-block suppression,
queue backpressure
Does not touch desktop/TUI code or add config flags. Plugin TTS provider
stream() metadata gap (#47896) is explicitly out of scope — built-in
ElevenLabs/OpenAI PCM streamers are the first consumers.
Refs: #60671, #47896
Follow-up to salvaged PR #56473: dedupe the five cleanup_*_cache bodies
into a shared _cleanup_cache_dir() helper, add cleanup_video_cache() and
cleanup_screenshot_cache() (with get_screenshot_cache_dir()), and drive
all five from a single (name, fn) loop in _start_gateway_housekeeping()
instead of one copy-pasted try/except per cache. Covers the video/
screenshot half of #56427.
Adds cleanup_audio_cache() to gateway/platforms/base.py and wires it into
_start_gateway_housekeeping() in gateway/run.py, matching the existing
image/document cache cleanup pattern.
Re-implements the fix proposed in #16473, which targeted the now-deprecated
_start_cron_ticker() and has since stalled with a merge conflict against
main. _start_gateway_housekeeping()'s docstring already claimed audio
cleanup ("prunes the image/audio/document cache"); this makes the code
match it.
Co-authored-by: nftpoetrist <264138787+nftpoetrist@users.noreply.github.com>
When WeChat (Weixin) returns a voice_item.text (Tencent Cloud's STT),
Hermes previously trusted that text as the user-visible message body and
skipped downloading the raw audio. For non-Chinese audio that text is
garbage — the original report was a Russian voice message that came
back as English phonemes — and the user sees nonsense as their own
message. International users on the WeChat gateway effectively can't
use voice.
Two short-circuits in gateway/platforms/weixin.py caused this:
- _download_voice() returned None whenever voice_item.text was set,
so the raw SILK/Opus audio was never fetched.
- _extract_text() returned voice_item.text verbatim as the body,
so even if the audio had been downloaded, the central STT
pipeline in gateway/run.py never had a chance to replace it.
Fix both: always download the raw audio (the central pipeline picks
it up via _collect_media()), and skip voice items in _extract_text()
so the body comes from Hermes' own mlx-whisper / whisper.cpp /
faster-whisper transcription instead of Tencent's. Behavior is
unchanged when voice_item.text is absent (the original happy path
where audio was already being downloaded).
Tests:
- 5 new tests in TestWeixinVoiceAlwaysDownloaded covering both
functions, the _collect_media integration path, and a
regression guard for the text-item path.
- 74/74 in tests/gateway/test_weixin.py pass.
Minimal fix: add 'if ct == "file": return False' before extension
matching. The original fallback logic is preserved for empty/unknown
content_types. Only the bug case (file uploads with audio extensions)
is fixed.
Removed the over-engineered _looks_like_voice helper.
Refined the fix: instead of removing extension-based fallback entirely,
only skip it when content_type is explicitly 'file' (or image/video).
Empty or unknown content_types still fall back to extension matching
as a defensive measure.
- content_type='voice' or 'audio/*' → True (API signal)
- content_type='file' → False (file transfer, never voice)
- content_type='' → extension fallback (defensive)
- content_type=unknown → extension fallback (defensive)
Added _looks_like_voice() module-level helper and comprehensive tests.
The _is_voice_content_type() heuristic matched audio file extensions
(.wav, .mp3, .ogg, etc.) even when the QQ Bot API explicitly reported
content_type='file'. This caused files sent via QQ's file-transfer
feature to be routed through the speech-to-text pipeline instead of
being saved as regular attachments.
The QQ Bot API already distinguishes voice messages (content_type=
'voice') from file uploads (content_type='file'), so filename-based
extension sniffing is unnecessary and harmful.
Removed the _VOICE_EXTENSIONS fallback; now only content_type is
checked.
Closes #XXXX
_invalidate_pending_stt_cache() clears the gateway-side transcription
cache when merge_pending_message_event() folds a follow-up message into a
still-pending event, so the next transcription picks up the merged text
and attachments. It also cleared _gateway_pending_stt_echo_sent, but that
flag is not derived state — it records that the transcript was already
delivered to the user.
Dropping it makes the re-run transcription echo the earlier notes a second
time. Both merge branches are affected, including the text-only follow-up
case where no new audio arrived at all: there the cache is invalidated,
the same voice note is transcribed again (a second paid STT call) and the
same line is echoed again.
Sequence:
1. voice note arrives, interrupt monitor transcribes it and echoes
'🎙️ "hello"'
2. user sends a follow-up while the turn is still pending, so it merges
3. drain path re-transcribes and echoes '🎙️ "hello"' a second time
Keep the ledger out of the invalidation set and track it as a count of
already-echoed transcripts instead of a single boolean. A count is what
the merge case actually needs: re-running transcription over the extended
media list returns the earlier transcripts as a prefix of the new one, so
echoing only the unsent tail suppresses the repeat while still surfacing a
newly merged voice note. A count rather than a set of seen values, so two
separate notes that transcribe identically stay two distinct deliveries —
covered by test_pending_stt_merge_echoes_two_identical_transcripts.
The guard stays within the 12-line window that
test_all_gateway_transcript_echo_sends_are_gated enforces over run.py.
[[audio_as_voice]] is message-global but was applied to every media file in a
message. A non-audio file flagged is_voice is excluded from the embedded-photo
batch and falls through to send_document, so an image in a message that also
carries a voice note arrives as a file attachment instead of an inline photo.
Gate the voice flag on the file extension so one message can carry an inline
image AND a voice bubble. Also wrap the path append in try/except so a crafted
~\x00 path is skipped rather than aborting extraction of all attachments.
Salvaged from PR #62040 (@giladbau), simplified per post-#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.
Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (#57049, #36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (#14841, #45557).
Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)
Fixes#57049Fixes#36685
Refs #14841#45557
When /reasoning show is enabled, the model's <think> blocks appear in
the final assistant message. TTS reads these aloud, which is unwanted —
users want to see reasoning but not hear it spoken.
- Add _THINK_BLOCK regex to _strip_markdown_for_tts() (tools/tts_tool.py)
— the general TTS text-preparation path used by all TTS providers
- Add <think> stripping to prepare_tts_text() (gateway/platforms/base.py)
— the gateway auto-TTS path
- Reuses the existing regex pattern from stream_tts_to_speaker()
Closes#34213
Review follow-up: the spoken script is for synthesis only. Caption
eligibility and payload stay on the original reply, so a long reply
whose normalized script fits the 1024 char limit is still delivered
in full as its own message. Adds the long-original/short-normalized
regression case to the auto-TTS caption tests.
Auto-TTS previously fed raw chat Markdown and compact symbols straight to the
speech provider, so units were read as stray letters and headings or bullets ran
together. This routes spoken text through a new normalizer,
tools/tts_text_normalize.prepare_spoken_text, that expands units (for example a
temperature written with the degree symbol becomes "degrees Celsius") and
flattens Markdown into a transcript-like script with sentence pauses.
The normalizer is best-effort: if it ever fails the code falls back to the
previous markdown-strip behavior, so auto-TTS keeps working. The Telegram voice
caption uses the same normalized text. Includes a unit test.
One sniffer owns magic-byte container detection (Teknium's one-concept-
one-owner rule): the new tools/audio_container.py is used by
- gateway/platforms/base.py _sniff_audio_ext (inbound cache — PR #36166's
central sniffer, now covering AAC/ADTS, MP4-brand disambiguation, webm)
- gateway/platforms/signal.py _guess_extension (audio/AV branches
delegated; RIFF/WAVE fix from PR #50690 and M4A-brand fix from
PR #72490 now live centrally)
- tools/tts_tool.py _sniff_audio_container (outbound repair, PR #73072)
cache_audio_from_url inherits the sniff via cache_audio_from_bytes.
Adds tests/tools/test_audio_container.py covering every magic-byte type,
wrong-extension repair on the inbound cache, unknown passthrough, the
URL path, and Signal's delegation.
iOS Signal delivers voice notes as MP4-container AAC carrying an audio
ftyp brand ("M4A "). `_guess_extension()` returned ".mp4" for every
`ftyp` file regardless of brand, so those attachments were cached as
documents instead of audio and STT rejected the upload:
API error: Error code: 400 - Invalid file format.
Supported formats: ['flac','m4a','mp3','mp4','mpeg','mpga','oga',
'ogg','wav','webm']
Read the 4-byte brand at offset 8 and return ".m4a" for audio brands
("M4A ", "M4B ") so they satisfy `_is_audio_ext()` and route to
`cache_audio_from_bytes()`. Video brands (isom/mp42/avc1/qt) still
return ".mp4".
This mirrors the existing brand/form-type disambiguation already used
in this function for RIFF (WEBP vs WAVE) and for ADTS AAC vs MP3.
Verified against a real iOS Signal voice note: pre-fix the upload was
rejected with the 400 above; post-fix the same bytes transcribe
successfully.
When FTS5 index corruption prevents INSERT INTO messages, the gateway
accumulates messages in _pending_messages (memory-only). On shutdown,
.clear() discards the only surviving copy — permanent user data loss.
Changes:
- Add gateway/shutdown_flush.py with flush_pending_to_file() and
recover_pending_to_db() for two-phase data preservation.
- Patch gateway/run.py: flush runner._pending_messages before clear()
in _stop_impl_body, and recover on startup after runner.start().
- Patch gateway/platforms/base.py: flush adapter._pending_messages
before clear() in the adapter shutdown path.
Recovery behavior:
- Reads pending JSON files from ~/.hermes/pending_messages/
- Inserts messages into state.db directly
- Per-session isolation: one corrupt session doesn't block others
- Successful recovery deletes the flush file
- Failed recovery re-saves for next startup retry
One relay adapter fronts N platforms on one WS, but the capability surface
(MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor
resolved the handshake — and the transport's read loop OVERWROTE it on every
descriptor frame (last-writer-wins). A Discord chat on a gateway whose
applied descriptor was Telegram's inherited the 4,096-char cap and over-sent
into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies
silently lost while inbound kept working).
- ws_transport: accumulate one descriptor per platform in
_descriptors_by_platform (exposed via descriptor_for_platform); the FIRST
descriptor of a connection generation stays the session default instead of
last-writer-wins; the map resets on re-dial.
- BasePlatformAdapter: new max_message_length_for_chat /
message_len_fn_for_chat hooks defaulting to the scalar surface (native
single-platform adapters unchanged).
- RelayAdapter: overrides resolve the chat's platform from _platform_by_chat
(the same map per-frame egress uses) and look up that platform's negotiated
descriptor; falls back to the scalar for unknown chats/transports.
- stream_consumer (streaming budget, _raw_message_limit, fallback-continuation
chunking) + run.py tool-progress limit now resolve per-chat.
Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified
fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream
consumer suites green (213 + 223).
Hardening on top of the salvaged #51642: free-text fields
(preview/goal/summary/output_tail) pass redact_sensitive_text(force=True)
before leaving on the public /v1/runs SSE stream — same treatment the API
already applies to error text — and child_session_id survives the
allowlist so clients can correlate the child's session. Both were flagged
in the sweeper review of #51642 and unaddressed.
Behind proxies like Cloudflare Warp that leave peer-initiated FIN in
CLOSE_WAIT, aiohttp's default 30s keepalive_timeout lets idle sockets
accumulate. Use keepalive_timeout=2 + enable_cleanup_closed=True on the
weixin SSL connector so idle connections drain promptly (#18451 class,
related #69089).
Partial salvage of #70939: only the weixin connector hardening survives;
the PR's event-loop-freeze watchdog half is superseded by the loop-liveness
watchdog already merged on main (selector floor + thread-based probe, config-
gated via gateway.loop_watchdog).
Cross-surface coverage #23768 missed (per review feedback):
- tools/clarify_gateway.py: _ClarifyEntry carries a multi_select flag
(register() accepts it; signature() exposes it to adapters).
_coerce_text_response now parses multi-select replies — comma- or
space-separated numbers ('1,3' / '1 3'), exact labels, dedup — into a
JSON array string that _parse_multi_select_response decodes into a
list. Out-of-range/unknown tokens reject the reply (native button UI)
or fall back to custom text (awaiting_text/'Other' mode).
- gateway/run.py: _clarify_callback_sync accepts multi_select and
registers it on the pending entry.
- gateway/platforms/base.py: default numbered-list text fallback tells
the user multiple selections are allowed and how to reply.
- tui_gateway/server.py: clarify_callback passes multi_select through
the clarify.request payload as a hint; renderers without checkbox
support ignore the field and remain single-select-compatible.
- tests: 13 new gateway tests (flag storage, comma/space/single-number
parsing, label matching, out-of-range rejection, dedup, end-to-end
resolve, single-select regressions).
Follow-up to the salvaged #71756: instead of webhook importing cron's
private _is_cron_silence_response, the loose autonomous-lane matcher now
lives in gateway/response_filters.py as is_autonomous_silence_response,
sharing LIVE_GATEWAY_SILENT_MARKERS with the interactive exact-marker
rule so the marker sets can never drift. Cron and webhook both delegate
to it. Interactive gateway behavior unchanged.
A webhook route that answered `[SILENT]` still delivered, whenever the model
added a sentence saying why it was staying quiet:
[SILENT]
The new inbound was the same email quoted back a second time, on a ticket
we already answered. Nothing new to reply to, so I closed it.
Webhook subscription prompts tell the agent to answer `[SILENT]` on a tick that
produced no story — a duplicate inbound, a stand-down because a sibling lane
already replied, a routine close. Nobody is waiting on the other end of a
webhook, so a "nothing happened" message has no reader.
Delivery went through the live gateway's `is_intentional_silence_response`,
which requires the response to be EXACTLY a marker. That rule is right for an
interactive chat: swallowing a real answer because it opens with a marker is
much worse than showing a stray marker. It is the wrong trade for an autonomous
lane, where a leaked non-story is a pointless notification on every tick and
models reliably append the explanation that flips the check back to "deliver".
Cron already resolved this the other way — `cron/scheduler.py` treats a marker
on its own first or last line as silence — so the two autonomous lanes
disagreed while the interactive path was fine.
Suppress in `WebhookAdapter.send`, before the deliver-type switch, so every
route (log, github_comment, cross-platform) behaves the same. Reuses cron's
`_is_cron_silence_response` rather than restating the rule, so the two lanes
cannot drift; prose that merely mentions a marker mid-sentence still delivers.
The interactive gateway path is untouched.
Tests: six cases in tests/gateway/test_webhook_adapter.py — bare marker,
marker + trailing prose (the reported shape), marker on the last line, a real
report, a report quoting a marker mid-sentence, and a `log` route. Verified
red-first: with the suppression removed the three silence cases fail
("Expected send to not have been awaited") while the three delivery cases still
pass, so the tests assert the fix rather than the framework.
Follow-up wave to #72170 resolving the remaining open MEDIA-delivery gaps:
- #24032: add .kmz/.kml/.geojson/.gpx to MEDIA_DELIVERY_EXTS, and recover
unknown-extension paths containing spaces via _match_extensionless_path —
validation-gated forward extension across single spaces (bounded at 8
tokens, stops at newline / next MEDIA: keyword). The regex itself stays
non-greedy and whitespace-bounded so the #68773 absorption bug class
cannot return; the on-disk file check is the oracle.
- #16434 (streaming half): _strip_media_tag_directives now uses the same
mask-as-locator pattern as extract_media, so MEDIA tags inside fenced
code blocks, inline-code examples, and JSON string values survive in
streamed display text instead of being mangled. Display and delivery
now agree on every protected-span rule.
- Updated three stream_consumer display expectations that pinned the old
inconsistent behavior (backtick/double-quote tags stripped from display
while delivery never attempted them).
The model sometimes echoes a previous MEDIA:path tag or bare file path in a later response. Previously both streaming delivery and non-streaming delivery would re-send those files, causing duplicate documents/images on later turns.
Changes:
- _collect_history_media_paths now also scans assistant messages for MEDIA: tags, not just tool results.
- _deliver_media_from_response accepts pre-computed history_media_paths and filters extracted media/local files.
- BasePlatformAdapter gains _history_media_paths_for_session for non-streaming dedup.
Two remaining formatting variants that silently killed file delivery:
- A trailing sentence period (MEDIA:/x/data.csv.) failed the boundary
lookahead, so the tag neither extracted nor stripped. The period is now
accepted as a boundary only when followed by whitespace/EOL, keeping
multi-part extensions (.tar.gz) intact.
- A whole tag wrapped in inline code (`MEDIA:/x/data.csv`) was masked as
a prose example (#35695). Models routinely format paths as inline code,
eating real deliveries. Inline-code tags now deliver when the path
validates on disk; non-existent example paths stay masked and fenced
code blocks remain fully masked.
Adds a regression matrix covering both plus the salvaged contributor
fixes (emphasis wrap, glued tags, glued [[as_document]], dedupe,
unknown-extension and extensionless delivery).
MEDIA_DELIVERY_EXTS in gateway/platforms/base.py omitted .3gp, causing
MEDIA: tags with .3gp files to leak as plain text instead of being
extracted for native video delivery. _VIDEO_EXTS in
tools/send_message_tool.py and _MIGRATION_VIDEO_EXTS in the Feishu
adapter omitted .webm, causing .webm files to be classified as
documents instead of video on Telegram and other platforms.
Both extensions are already present in every gateway-side _VIDEO_EXTS
definition (run.py, kanban_watchers.py, weixin.py, base.py local).
Closes#71621, Closes#71603
Surface a user-visible delivery notice when non-streaming media dispatch
gets success=False after MEDIA tags were stripped, and validate forum
starter-message attachments the same way as direct channel sends (#66797).
Outbound MEDIA video/document tags on the Discord non-streaming path
were extracted (stripped from the visible text) but never delivered:
no attachment, no error, and no dispatch log line (#66797). The
open-handle discord.File plus singular file= form could race Discord's
multipart encoder after an earlier image batch on the same channel and
return a successful message carrying zero attachments.
Fix _send_file_attachment (used by send_video/send_document/
send_image_file) to:
- use a path-based discord.File via the plural files=[...] kwarg, the
same pattern as the working send_multiple_images batch path;
- pre-flight os.path.isfile and return a File not found result instead
of raising deep inside discord.File;
- fail loud when Discord accepts the message but attaches nothing, so
the dispatch loop surfaces a warning rather than a silent drop;
- add INFO dispatch logs (base non-image MEDIA fan-out, video send, and
file attachment) so a MEDIA:.mp4 cannot vanish without a trace.
Tests (tests/gateway/test_discord_send.py):
- path-based files=[...] kwarg used, singular file= unset;
- fail-loud when the returned message has no attachments;
- missing file fails fast without resolving the channel;
- forum-parent delivery routes through create_thread with files=[...];
- end-to-end image+video response routes the mp4 to send_video while
images still batch.
Fixes#66797
When the same file is referenced multiple times in one message (common
when the agent emits MEDIA tags both inline and in a summary footer),
the platform adapter uploads/sends the same file twice — visible as
duplicate attachments in Telegram, duplicate posts in Slack, etc.
Set-based dedup on the expanded path, preserving first-occurrence order.
Two MEDIA: path tags emitted back-to-back without a separator merged
into a single invalid path and were silently dropped. The same happened
for extension-less tags.
Root cause: both regexes in gateway/platforms/base.py used greedy
quantifiers in their path class, causing adjacent tags to be absorbed.
Fix: make both regexes non-greedy (add ? to quantifiers) and add
MEDIA: to the trailing lookahead boundary set so the next MEDIA:
keyword stops the current match cleanly.
Models routinely present a file to the user with the delivery tag wrapped in
Markdown emphasis — `**MEDIA:/path.pptx**`, `*MEDIA:/path*`, `_MEDIA:/path_`.
MEDIA_TAG_CLEANUP_RE only tolerated a single leading/trailing quote or backtick
(`[`"']?`), and its closing lookahead set excluded `*` and `_`, so an
emphasis-wrapped tag never matched. The file was then silently never delivered
and the literal `MEDIA:/path` text leaked into the chat instead — the user sees
a path, not the attachment.
Allow a short run of emphasis/quote markers (`[`"'*_]{0,3}`) on both sides of
the tag and add `*`/`_` to the closing lookahead. Code-block, inline-code and
blockquote contexts are still neutralised earlier by `_mask_protected_spans`
(#35695), so documentation/example tags remain non-deliverable; the
absolute-path anchor still rejects relative paths; `_` inside a filename is
unaffected.
Adds regression coverage in TestExtractMedia for bold/italic/underscore
wrapping, mid-prose bold, emphasis-wrapped .html, underscore-in-filename, and
emphasis-wrapped relative-path rejection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MEDIA_TAG_CLEANUP_RE failed to match when a directive like [[as_document]]
was concatenated directly to the file extension without whitespace
(e.g., MEDIA:/home/user/report.xlsx[[as_document]]). This caused the
file to be silently not delivered while the gateway reported success.
Root cause: the lookahead character class [\s`",;:)\}\]|$) did not
include [, so [[as_document]] immediately after the extension broke
the lookahead assertion.
Fix: add \[ to the lookahead class so directives can follow the path
without whitespace. This is safe because [ is already stripped elsewhere
in the same file via .replace("[[as_document]]", "").
Added regression tests covering:
- Directive glued to extension (issue case)
- Directive with whitespace (baseline)
- Tag at end of string ($ anchor)
- RelayAdapter.send_exec_approval / send_slash_confirm / send_clarify:
override the base text fallbacks with ONE platform-abstract `prompt` op
(connector renders Discord components / Telegram inline keyboards /
Slack Block Kit / WhatsApp buttons+lists). Option sets mirror the native
adapters exactly (once/session/always/deny with the same
allow_session/allow_permanent/smart_denied gating; once/always/cancel;
choices + Other). Clarify option ids are positional (c0..cN/other) —
choice text is arbitrary UTF-8, callback budgets are 64 bytes.
- Pending-prompt registry: gateway-minted 8-hex prompt ids →
{kind, session_key, extras}; one answer wins, lazy expiry, unanswered
prompts swept opportunistically. Wire timeout_s stays advisory.
- _consume_prompt_response (wired into _on_inbound AND the Discord
passthrough lane): routes answers to the SAME primitives the native
button handlers call — tools.approval.resolve_gateway_approval,
tools.slash_confirm.resolve, tools.clarify_gateway
resolve/mark_awaiting_text — then acks in-channel. Unknown/expired ids
fall through as command-shaped text (typed-reply degradation, the
relay's analog of the native 'approval expired' edit).
- Discord type-3 stub replaced: an hp1:<prompt>:<option> custom_id decodes
to a structured prompt_response (codec mirrored from the connector's
promptCodec); foreign custom_ids keep the legacy best-effort text shape.
- MessageEvent.prompt_response field + ws_transport wire parsing (additive).
- react ack lifecycle: on_processing_start/complete → `react` ops
(👀 → ✅/❌, remove-then-add), op-gated on supported_ops, best-effort by
contract (a react failure never touches the turn).
- Op gating throughout: a connector not advertising `prompt` gets
success=False from send_exec_approval/send_slash_confirm (run.py's text
fallback takes over — same contract as a failed native button send) and
the base numbered-text clarify; `react` silently no-ops.
- docs/relay-connector-contract.md §4: prompt / prompt_response / react
semantics (callback token, budgets, authorization-parity, foreign-id
behavior, per-platform react mappings).
- tests: tests/gateway/relay/test_relay_interactive.py (19) — option-set
rendering + gating matrices, registry consume-once/expiry, resolver
routing for all three kinds (monkeypatched primitives), fall-through
cases, Discord hp1 decode + foreign-id shape, react lifecycle
(success/failure/cancelled), op-gated/best-effort react.
Cross-repo pair: gateway-gateway 'Phase 3 interactive' PR (prompt/react
senders on all four lanes + interaction ingest).
Add a persisted, backend-confirmed provider/model lock for Hermes
Browser and other session API clients. A confirmed lock is an
execution contract rather than response metadata:
- POST /api/sessions/{session_id}/model validates and persists a
confirmed browser_model_lock (advertised in /v1/capabilities)
- session chat + chat/stream consume the persisted lock on body-only
follow-up turns; a confirmed lock wins over an older gateway session
/model override and the session-persisted model
- a later successful session /model switch explicitly clears and
replaces the lock while preserving lineage markers (_branched_from)
and invalidating cached system-prompt model/provider metadata
- ordinary one-off request overrides never replace a confirmed lock
- provider-resolution failure fails closed as a typed provider-auth
error (controlled response, never global-credential reuse)
- confirmed locks disable the global fallback model chain
- the completed agent's actual provider/model must match the locked
route or the turn fails with a runtime-mismatch error
- responses carry sanitized runtime metadata reporting actual vs
requested provider/model and lock state
Rebased onto the provider-aware request routing (#70853) and
session-model parity (#70931) that landed since the original branch;
the lock now slots into that precedence chain as the top rung.
Salvaged from PR #61236 by @abundantbeing.
Three parity fixes between the API server and the native gateway's
agent-runtime resolution, integrated with the provider-aware request
routing that landed in #70853:
- Session-persisted model is honored: POST /api/sessions {"model": ...}
stores a model that the chat handlers previously fetched and threw
away. A stored value that matches a model_routes alias goes through
the route path (route provider/credentials apply); a raw model string
threads through as session_model, pinning the session's turns ahead
of per-request body values but below an explicit session /model
override.
- Empty-model recovery: provider-catalog default when config has no
model.default but a provider resolved, plus last-known-good model
recovery (#35314) keyed on gateway_session_key only (never ephemeral
session_id — no unbounded growth from one-off requests).
- Provider auth failures surface as controlled responses: RuntimeError
from _resolve_runtime_agent_kwargs() is re-raised as a dedicated
_ProviderAuthResolutionError at the call site, caught narrowly in
_run_agent() and the /v1/runs executor to return run.py's response
shape instead of an undifferentiated 500 (session-chat endpoints
previously returned a raw aiohttp 500 with no JSON body).
Salvaged from PR #57947 by @FvanW; session-model route-alias resolution
from PR #59941 by @kaishi00.
Co-authored-by: kaishi00 <kaishi00@users.noreply.github.com>
