When WeChat (Weixin) returns a voice_item.text (Tencent Cloud's STT),
Hermes previously trusted that text as the user-visible message body and
skipped downloading the raw audio. For non-Chinese audio that text is
garbage — the original report was a Russian voice message that came
back as English phonemes — and the user sees nonsense as their own
message. International users on the WeChat gateway effectively can't
use voice.
Two short-circuits in gateway/platforms/weixin.py caused this:
- _download_voice() returned None whenever voice_item.text was set,
so the raw SILK/Opus audio was never fetched.
- _extract_text() returned voice_item.text verbatim as the body,
so even if the audio had been downloaded, the central STT
pipeline in gateway/run.py never had a chance to replace it.
Fix both: always download the raw audio (the central pipeline picks
it up via _collect_media()), and skip voice items in _extract_text()
so the body comes from Hermes' own mlx-whisper / whisper.cpp /
faster-whisper transcription instead of Tencent's. Behavior is
unchanged when voice_item.text is absent (the original happy path
where audio was already being downloaded).
Tests:
- 5 new tests in TestWeixinVoiceAlwaysDownloaded covering both
functions, the _collect_media integration path, and a
regression guard for the text-item path.
- 74/74 in tests/gateway/test_weixin.py pass.
Minimal fix: add 'if ct == "file": return False' before extension
matching. The original fallback logic is preserved for empty/unknown
content_types. Only the bug case (file uploads with audio extensions)
is fixed.
Removed the over-engineered _looks_like_voice helper.
Refined the fix: instead of removing extension-based fallback entirely,
only skip it when content_type is explicitly 'file' (or image/video).
Empty or unknown content_types still fall back to extension matching
as a defensive measure.
- content_type='voice' or 'audio/*' → True (API signal)
- content_type='file' → False (file transfer, never voice)
- content_type='' → extension fallback (defensive)
- content_type=unknown → extension fallback (defensive)
Added _looks_like_voice() module-level helper and comprehensive tests.
The _is_voice_content_type() heuristic matched audio file extensions
(.wav, .mp3, .ogg, etc.) even when the QQ Bot API explicitly reported
content_type='file'. This caused files sent via QQ's file-transfer
feature to be routed through the speech-to-text pipeline instead of
being saved as regular attachments.
The QQ Bot API already distinguishes voice messages (content_type=
'voice') from file uploads (content_type='file'), so filename-based
extension sniffing is unnecessary and harmful.
Removed the _VOICE_EXTENSIONS fallback; now only content_type is
checked.
Closes #XXXX
Trimmed cherry-pick of PR #40592 (duration + dedup hunks only; the
voice-classification hunk duplicates #29235 and the send_voice Opus
rewrite is out of scope for this inbound-focused PR):
- adapter.py: ffprobe duration (off-loop) attached to Feishu voice
uploads via _build_file_upload_body(duration=...) and the audio
message payload (#16524, #8300)
- gateway/run.py: TTS dedup narrowed to the current turn;
_enrich_message_with_transcription dedups repeated audio paths
Refs #40592#16524#8300
Two log-spam bugs found in live gateway logs:
1. gateway_routing UNIQUE-constraint spam (261 warnings in one errors.log):
early builds of the #59203 routing-index migration created
gateway_routing with 'session_key TEXT PRIMARY KEY' and no scope
column. _reconcile_columns() ADDs the missing scope column but SQLite
cannot ALTER a primary key, so the shipped composite
PRIMARY KEY (scope, session_key) never lands on those databases. Both
write paths then fail on every save:
- save_gateway_routing_entry: 'ON CONFLICT clause does not match any
PRIMARY KEY or UNIQUE constraint'
- replace_gateway_routing_entries: 'UNIQUE constraint failed:
gateway_routing.session_key' whenever the same session_key exists
under another scope (e.g. test-suite scopes leaked into a live DB).
New _heal_gateway_routing_pk() rebuilds the table once with the
composite key, preserving rows (newest wins on collisions, NULL scope
coalesced to ''). Same one-time-heal pattern as the #51646 active-
column repair. Verified E2E against a copy of a real affected state.db.
2. pre_gateway_dispatch warned ''GatewayRunner' object has no attribute
'session_store'' and silently dropped the hook for every message on
partially-initialized runners (bare object.__new__ runners in tests,
and any future init-order change). Pass
getattr(self, 'session_store', None) so the hook always fires
(pitfall #17 pattern).
Both regression tests fail without their fixes (sabotage-verified).
Follow-up for salvaged #65023/#53020: _prepare_busy_steer_text now calls
_transcribe_and_echo_pending_voice (the same helper the interrupt monitor
and pending-drain paths use) instead of a private transcription+echo copy,
so out-of-band voice pays one STT call per platform message and the echo
respects the count-based ledger from #67281. can_steer now accepts events
whose attachments are all STT-eligible voice media, completing the steer
half of #58780. Adds extract_media gating tests for #44826 and the
contributor mapping for chefboyrdave21.
_invalidate_pending_stt_cache() clears the gateway-side transcription
cache when merge_pending_message_event() folds a follow-up message into a
still-pending event, so the next transcription picks up the merged text
and attachments. It also cleared _gateway_pending_stt_echo_sent, but that
flag is not derived state — it records that the transcript was already
delivered to the user.
Dropping it makes the re-run transcription echo the earlier notes a second
time. Both merge branches are affected, including the text-only follow-up
case where no new audio arrived at all: there the cache is invalidated,
the same voice note is transcribed again (a second paid STT call) and the
same line is echoed again.
Sequence:
1. voice note arrives, interrupt monitor transcribes it and echoes
'🎙️ "hello"'
2. user sends a follow-up while the turn is still pending, so it merges
3. drain path re-transcribes and echoes '🎙️ "hello"' a second time
Keep the ledger out of the invalidation set and track it as a count of
already-echoed transcripts instead of a single boolean. A count is what
the merge case actually needs: re-running transcription over the extended
media list returns the earlier transcripts as a prefix of the new one, so
echoing only the unsent tail suppresses the repeat while still surfacing a
newly merged voice note. A count rather than a set of seen values, so two
separate notes that transcribe identically stay two distinct deliveries —
covered by test_pending_stt_merge_echoes_two_identical_transcripts.
The guard stays within the 12-line window that
test_all_gateway_transcript_echo_sends_are_gated enforces over run.py.
[[audio_as_voice]] is message-global but was applied to every media file in a
message. A non-audio file flagged is_voice is excluded from the embedded-photo
batch and falls through to send_document, so an image in a message that also
carries a voice note arrives as a file attachment instead of an inline photo.
Gate the voice flag on the file extension so one message can carry an inline
image AND a voice bubble. Also wrap the path append in try/except so a crafted
~\x00 path is skipped rather than aborting extraction of all attachments.
Two hand-written fixes in the voice input path:
- _handle_voice_channel_input now resolves the bound text channel's
channel_prompt via the adapter's _resolve_channel_prompt so voice input
gets the same per-channel context as typed messages (fixes#50149).
- _enrich_message_with_transcription now guards success=True results whose
transcript is empty/whitespace-only (silence, cut-off, inaudible audio):
instead of emitting empty quotes the agent gets a clear sentinel note.
Reimplemented against the current plain-quoted note wording; original
concept and tests by @deacon-botdoctor in PR #41603 (fixes#41603).
- Wire adapter._voice_input_callback at connect and reconnect so voice
transcription is forwarded without requiring /voice join (#60623).
- Add optional text_channel_id and source params to DiscordAdapter
.join_voice_channel() so automatic/programmatic voice joins can
establish the text-channel binding needed by _handle_voice_channel_input.
- Add TestVoiceInputCallbackWiring: asserts callback wiring on startup
and reconnect for Discord adapters with voice attributes.
_voice_input_callback was only set in _handle_voice_channel_join, not at adapter connect or reconnect. Voice transcription was logged but never forwarded as an inbound message without explicit /voice join.
Wire the callback at both connect and reconnect paths.
Fixes#60623
Salvaged from PR #62040 (@giladbau), simplified per post-#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.
Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (#57049, #36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (#14841, #45557).
Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)
Fixes#57049Fixes#36685
Refs #14841#45557
Salvaged from PR #51196 (@55nx954gn6-debug). _should_send_voice_reply only
consulted the runner's _voice_mode dict (/voice on|voice_only|all), so the
global voice.auto_tts config default — which is synced into each adapter's
_auto_tts_default on gateway connect — was invisible to the runner path.
Net effect: with streaming enabled and only global auto-TTS configured
(no per-chat /voice opt-in), the streamed reply consumed the text, the
base adapter's auto-TTS got text_content=None, and no voice reply was
ever sent (#51867/#23983 remainder).
The runner now also asks the adapter's _should_auto_tts_for_chat(chat_id)
(which encodes per-chat /voice on|off overrides over the global default);
an explicit /voice off chat mode remains a hard override.
Refs #51867#23983#51282#13126
When /reasoning show is enabled, the model's <think> blocks appear in
the final assistant message. TTS reads these aloud, which is unwanted —
users want to see reasoning but not hear it spoken.
- Add _THINK_BLOCK regex to _strip_markdown_for_tts() (tools/tts_tool.py)
— the general TTS text-preparation path used by all TTS providers
- Add <think> stripping to prepare_tts_text() (gateway/platforms/base.py)
— the gateway auto-TTS path
- Reuses the existing regex pattern from stream_tts_to_speaker()
Closes#34213
Review follow-up: the spoken script is for synthesis only. Caption
eligibility and payload stay on the original reply, so a long reply
whose normalized script fits the 1024 char limit is still delivered
in full as its own message. Adds the long-original/short-normalized
regression case to the auto-TTS caption tests.
Auto-TTS previously fed raw chat Markdown and compact symbols straight to the
speech provider, so units were read as stray letters and headings or bullets ran
together. This routes spoken text through a new normalizer,
tools/tts_text_normalize.prepare_spoken_text, that expands units (for example a
temperature written with the degree symbol becomes "degrees Celsius") and
flattens Markdown into a transcript-like script with sentence pauses.
The normalizer is best-effort: if it ever fails the code falls back to the
previous markdown-strip behavior, so auto-TTS keeps working. The Telegram voice
caption uses the same normalized text. Includes a unit test.
One sniffer owns magic-byte container detection (Teknium's one-concept-
one-owner rule): the new tools/audio_container.py is used by
- gateway/platforms/base.py _sniff_audio_ext (inbound cache — PR #36166's
central sniffer, now covering AAC/ADTS, MP4-brand disambiguation, webm)
- gateway/platforms/signal.py _guess_extension (audio/AV branches
delegated; RIFF/WAVE fix from PR #50690 and M4A-brand fix from
PR #72490 now live centrally)
- tools/tts_tool.py _sniff_audio_container (outbound repair, PR #73072)
cache_audio_from_url inherits the sniff via cache_audio_from_bytes.
Adds tests/tools/test_audio_container.py covering every magic-byte type,
wrong-extension repair on the inbound cache, unknown passthrough, the
URL path, and Signal's delegation.
iOS Signal delivers voice notes as MP4-container AAC carrying an audio
ftyp brand ("M4A "). `_guess_extension()` returned ".mp4" for every
`ftyp` file regardless of brand, so those attachments were cached as
documents instead of audio and STT rejected the upload:
API error: Error code: 400 - Invalid file format.
Supported formats: ['flac','m4a','mp3','mp4','mpeg','mpga','oga',
'ogg','wav','webm']
Read the 4-byte brand at offset 8 and return ".m4a" for audio brands
("M4A ", "M4B ") so they satisfy `_is_audio_ext()` and route to
`cache_audio_from_bytes()`. Video brands (isom/mp42/avc1/qt) still
return ".mp4".
This mirrors the existing brand/form-type disambiguation already used
in this function for RIFF (WEBP vs WAVE) and for ADTS AAC vs MP3.
Verified against a real iOS Signal voice note: pre-fix the upload was
rejected with the 400 above; post-fix the same bytes transcribe
successfully.
The cap is shared across CLI, desktop/TUI and the messaging gateway, so the
surface that gets rejected is rarely the one holding the slots. The rejection
read "Hermes is at the active session limit (5/5). Try again when another
session finishes." while every slot was an idle desktop tab, which took
filesystem access to work out.
Name the holders in the message, and show slot usage plus each holder in
`hermes status`. Both are inert when max_concurrent_sessions is unset, which
is the default. The gateway's duplicate copy of the message now reuses the
shared helper.
Manual /compress already persists the rotated child transcript first and
only then repoints the live session_entry; a False rewrite_transcript
return keeps the entry on the original session_id so the conversation
stays reachable. Session hygiene auto-compress did the opposite: it
rebound session_id (and lease/topic) first, then called rewrite_transcript
without checking the return value. On a failed write the live entry
already pointed at an empty child SID and the turn continued — permanent
silent conversation loss. Persist first; rebind only after success.
list_recent_user_messages and the in-memory /retry + session.undo walkers
treated every role=user row as a real user turn. Timeline bookkeeping
(model_switch, async_delegation_complete, auto_continue, hidden) is stored
that way, so /undo soft-deleted from a marker and /retry re-sent opaque
bookkeeping text. Exclude display_kind the same way CLI resume counting and
the prompt.submit ordinal path do.
Critical fixes to salvaged PR #73020:
- Use SessionDB.append_message instead of raw INSERT INTO messages.
The original used wrong column names (session_key/created_at vs
session_id/timestamp) and bypassed FTS indexing, session metadata
updates, display_kind, and all other columns append_message handles.
- Use get_hermes_home() instead of hardcoded Path.home()/'.hermes'.
Profile-aware path resolution under HERMES_HOME override and active
profile isolation.
- Add 11 tests covering flush, recovery, serialisation, edge cases.
When FTS5 index corruption prevents INSERT INTO messages, the gateway
accumulates messages in _pending_messages (memory-only). On shutdown,
.clear() discards the only surviving copy — permanent user data loss.
Changes:
- Add gateway/shutdown_flush.py with flush_pending_to_file() and
recover_pending_to_db() for two-phase data preservation.
- Patch gateway/run.py: flush runner._pending_messages before clear()
in _stop_impl_body, and recover on startup after runner.start().
- Patch gateway/platforms/base.py: flush adapter._pending_messages
before clear() in the adapter shutdown path.
Recovery behavior:
- Reads pending JSON files from ~/.hermes/pending_messages/
- Inserts messages into state.db directly
- Per-session isolation: one corrupt session doesn't block others
- Successful recovery deletes the flush file
- Failed recovery re-saves for next startup retry
One relay adapter fronts N platforms on one WS, but the capability surface
(MAX_MESSAGE_LENGTH / message_len_fn) was a scalar from whichever descriptor
resolved the handshake — and the transport's read loop OVERWROTE it on every
descriptor frame (last-writer-wins). A Discord chat on a gateway whose
applied descriptor was Telegram's inherited the 4,096-char cap and over-sent
into Discord's 2,000-char API 400 (observed live: 2,543/2,641-char replies
silently lost while inbound kept working).
- ws_transport: accumulate one descriptor per platform in
_descriptors_by_platform (exposed via descriptor_for_platform); the FIRST
descriptor of a connection generation stays the session default instead of
last-writer-wins; the map resets on re-dial.
- BasePlatformAdapter: new max_message_length_for_chat /
message_len_fn_for_chat hooks defaulting to the scalar surface (native
single-platform adapters unchanged).
- RelayAdapter: overrides resolve the chat's platform from _platform_by_chat
(the same map per-frame egress uses) and look up that platform's negotiated
descriptor; falls back to the scalar for unknown chats/transports.
- stream_consumer (streaming budget, _raw_message_limit, fallback-continuation
chunking) + run.py tool-progress limit now resolve per-chat.
Tests: tests/gateway/relay/test_relay_per_platform_caps.py (7) — verified
fail-without/pass-with (all 7 fail with the fix stashed). Relay + stream
consumer suites green (213 + 223).
Hardening on top of the salvaged #51642: free-text fields
(preview/goal/summary/output_tail) pass redact_sensitive_text(force=True)
before leaving on the public /v1/runs SSE stream — same treatment the API
already applies to error text — and child_session_id survives the
allowlist so clients can correlate the child's session. Both were flagged
in the sweeper review of #51642 and unaddressed.
Behind proxies like Cloudflare Warp that leave peer-initiated FIN in
CLOSE_WAIT, aiohttp's default 30s keepalive_timeout lets idle sockets
accumulate. Use keepalive_timeout=2 + enable_cleanup_closed=True on the
weixin SSL connector so idle connections drain promptly (#18451 class,
related #69089).
Partial salvage of #70939: only the weixin connector hardening survives;
the PR's event-loop-freeze watchdog half is superseded by the loop-liveness
watchdog already merged on main (selector floor + thread-based probe, config-
gated via gateway.loop_watchdog).
Follow-up to the salvaged #71867 supervision fix. _spawn_supervised's own
backoff respawn created a new task without updating the external handle
self._reconnect_watcher_task, so after the reconnect watcher crashed and
self-restarted, _ensure_reconnect_watcher_running() saw the stale handle as
done() and spawned a SECOND concurrent watcher (double reconnect attempts).
Add an optional on_spawn callback to _spawn_supervised, fired with the live
task on every spawn INCLUDING internal respawns, and pass it at both reconnect-
watcher spawn sites so the tracked handle always advances. The two supervision
mechanisms (supervisor auto-restart + ensure-respawn) now compose instead of
racing. Regression test sabotage-verified.
Fixes#71758.
A platform adapter that dies on a transient upstream failure (marked
retryable=True, e.g. photon's sidecar exiting when its upstream
gRPC/CDN returns errors) is correctly queued into _failed_platforms
for background reconnection. But the reconnect watcher task itself
was spawned via a bare asyncio.create_task -- if an exception ever
escaped its OUTER while-loop (not just the per-platform inner
try/except), the watcher died silently: no log, no restart.
_ensure_reconnect_watcher_running() already existed to respawn a dead
watcher, but it's only called from _handle_adapter_fatal_error_impl()
when a NEW platform's fatal error arrives. If the watcher dies while a
platform is already sitting in the queue and no OTHER platform ever
fails afterward, nothing ever notices the watcher is dead -- exactly
matching the reported symptom: photon queued for reconnect, the
gateway itself healthy (other platforms kept working, so nothing
re-triggered the ensure-alive check), and the platform stayed dead for
17.5h until a manual restart, well after the transient upstream outage
had recovered.
Fix: spawn the reconnect watcher via the existing _spawn_supervised()
task-level supervisor (already used for kanban_dispatcher_watcher,
handoff_watcher, etc.) instead of a bare asyncio.create_task, at both
the initial startup spawn and the manual-respawn path in
_ensure_reconnect_watcher_running(). _spawn_supervised already
provides exactly what's missing here: catches and logs any exception
escaping the task, and auto-restarts with capped exponential backoff
(healthy-run counter resets so a daemon that crashes occasionally over
days is never permanently abandoned) -- self-healing independent of
any new fatal-error event.
Also hardened a related race: the watcher's per-platform loop looked
up self._failed_platforms[platform] via direct indexing after
snapshotting the keys with list(...). A platform removed concurrently
between the snapshot and the lookup (e.g. a manual /platform resume,
or a reconnect that succeeded via a different path) would raise an
uncaught KeyError -- exactly the class of bug this fix's supervision
now catches, but avoiding the crash-and-restart cycle entirely is
better than relying on it. Changed to .get() with a skip-if-missing
guard.
6 new tests pass (initial spawn uses _spawn_supervised, manual respawn
uses _spawn_supervised, the core regression -- watcher self-heals
after an uncaught exception with no new fatal-error event -- and the
race-guard scenario); 52/52 in the full
tests/gateway/test_platform_reconnect.py file (including the 4
pre-existing _ensure_reconnect_watcher_running tests, confirming no
regression to that respawn-when-dead-or-missing behavior).
