Two hand-written fixes in the voice input path:
- _handle_voice_channel_input now resolves the bound text channel's
channel_prompt via the adapter's _resolve_channel_prompt so voice input
gets the same per-channel context as typed messages (fixes#50149).
- _enrich_message_with_transcription now guards success=True results whose
transcript is empty/whitespace-only (silence, cut-off, inaudible audio):
instead of emitting empty quotes the agent gets a clear sentinel note.
Reimplemented against the current plain-quoted note wording; original
concept and tests by @deacon-botdoctor in PR #41603 (fixes#41603).
Keep one owner for PATH/local-prefix ffmpeg discovery: ffmpeg_utils now
delegates to tools.transcription_tools._find_ffmpeg_binary and only adds
the Discord-specific FFMPEG_PATH override and Windows winget fallback on
top (follow-up to PR #60627 by @LauraGPT, fixes#60624).
Discord's voice socket needs a brief warm-up before receiving clients
actually hear audio; the first ~100-200ms is lost, clipping the first
word/syllable of TTS playback. Prepend a configurable lead of silence to
speech on both playback paths:
- Mixer path: new _lead_silence_bytes() helper prepends PCM silence
(BYTES_PER_MS constant added to voice_mixer.py) before play_speech, on
both the reply and the pre-tool ack.
- Legacy FFmpegPCMAudio path: apply -af adelay=<ms>:all=1.
Tunable via discord.voice_fx.lead_silence_ms (default 200, 0 disables).
Fixes#66827
VoiceMixer duck-typed the discord.AudioSource interface (is_opus, read,
cleanup) but never inherited from it. discord.py's vc.play() does an
isinstance check and rejects non-AudioSource objects, causing the voice
fx mixer to silently fail with:
"Voice mixer failed to start: source must be an AudioSource not
VoiceMixer"
Added missing `import discord` and changed the class definition from
`class VoiceMixer:` to `class VoiceMixer(discord.AudioSource):`.
- Wire adapter._voice_input_callback at connect and reconnect so voice
transcription is forwarded without requiring /voice join (#60623).
- Add optional text_channel_id and source params to DiscordAdapter
.join_voice_channel() so automatic/programmatic voice joins can
establish the text-channel binding needed by _handle_voice_channel_input.
- Add TestVoiceInputCallbackWiring: asserts callback wiring on startup
and reconnect for Discord adapters with voice attributes.
PR #61407 accesses self._handle_voice_channel_input in _platform_reconnect_watcher. The test mock runner created via _make_runner() must have this attribute.
_voice_input_callback was only set in _handle_voice_channel_join, not at adapter connect or reconnect. Voice transcription was logged but never forwarded as an inbound message without explicit /voice join.
Wire the callback at both connect and reconnect paths.
Fixes#60623
- tests/gateway/test_base_auto_tts_output_format.py (new): the base
adapter auto-TTS block passes an explicit .ogg output path on every
OPUS_VOICE_PLATFORMS member (parametrized from the tts_tool set — the
single source of truth), keeps .mp3 on non-opus platforms, honors the
tool's success flag, and stays unique/uuid-based.
- tests/gateway/test_auto_voice_reply_format.py: runner _send_voice_reply
parametrized across Matrix/Feishu/WhatsApp/Signal (.ogg) alongside the
existing Telegram/Slack cases; streamed+global-auto-TTS gate regression.
- tests/gateway/test_telegram_voice_caption_markdown.py (new): caption
MarkdownV2 formatting, entity-rejection plain fallback, overflow skip,
and no-caption passthrough (#32029).
- test_voice_command.py filename-uuid contract test updated to point at
build_auto_tts_output_path (construction moved there).
Closes the #32029 gap: TelegramAdapter.send_voice passed captions raw with
no parse_mode, so auto-TTS captions (which carry the agent's markdown
reply) showed literal *asterisks*, backticks and [links](...). Captions
are now formatted to MarkdownV2 via the adapter's format_message when the
formatted text fits Telegram's 1024-char caption cap, with fallback to the
plain truncated caption when formatting overflows or the Bot API rejects
the entities (mirrors the text-send markdown fallback ladder).
Fixes#32029
Salvaged from PR #62040 (@giladbau), simplified per post-#73072 main: the
central _repair_ogg_container transcode makes an explicit .ogg output path
sufficient — no target_platform plumbing through the TTS tool needed.
Root cause (class-level): both gateway auto-TTS delivery call sites relied
on the TTS tool reading HERMES_SESSION_PLATFORM to pick Ogg/Opus vs MP3,
but that contextvar is cleared by _clear_session_env before the base
adapter's post-handler auto-TTS block runs, so want_opus was always False
on that path → MP3 → Telegram sent an audio attachment instead of a native
voice bubble (#57049, #36685). The runner's _send_voice_reply had the
sibling bug: it hardcoded .ogg for Telegram only, leaving Matrix and
Feishu runner voice replies as MP3 (#14841, #45557).
Fix: new build_auto_tts_output_path(platform) in gateway/platforms/base.py
hands an explicit .ogg temp path when the platform is in the TTS tool's
OPUS_VOICE_PLATFORMS set (single source of truth — telegram/matrix/feishu/
whatsapp/signal), .mp3 otherwise. Used by BOTH delivery call sites:
- BasePlatformAdapter auto-TTS block (also honors the tool's success flag
and cleans up requested + returned paths)
- GatewayRunner._send_voice_reply (replaces the telegram-only ternary)
Fixes#57049Fixes#36685
Refs #14841#45557
Salvaged from PR #51196 (@55nx954gn6-debug). _should_send_voice_reply only
consulted the runner's _voice_mode dict (/voice on|voice_only|all), so the
global voice.auto_tts config default — which is synced into each adapter's
_auto_tts_default on gateway connect — was invisible to the runner path.
Net effect: with streaming enabled and only global auto-TTS configured
(no per-chat /voice opt-in), the streamed reply consumed the text, the
base adapter's auto-TTS got text_content=None, and no voice reply was
ever sent (#51867/#23983 remainder).
The runner now also asks the adapter's _should_auto_tts_for_chat(chat_id)
(which encodes per-chat /voice on|off overrides over the global default);
an explicit /voice off chat mode remains a hard override.
Refs #51867#23983#51282#13126
Salvaged from PR #53157 (@LLQWQ). Three changes for native Feishu voice
bubble delivery:
1. Include audio duration in the file-upload body when uploading opus
files — Feishu renders 0:00 bubbles without it. Duration is extracted
by parsing the OGG container's last granule position (pure Python,
no ffprobe dependency).
2. Thread routing fallback: Feishu's create-message API rejects
msg_type='audio' with receive_id_type='thread_id' (error 99992402);
retry via the reply API against the thread's last message, then fall
back to chat_id routing.
3. (dropped) tools/tts_tool.py want_opus hunk — superseded by main's
OPUS_VOICE_PLATFORMS, which already includes feishu. The PR's stray
scripts/release.py hunk was also dropped (frozen AUTHOR_MAP policy;
mapping added under contributors/emails/ instead).
Fixes#45557
Refs #18831#16524
Salvaged from PR #18861 (@shellybotmoyer). Switch the central Opus
transcoders from CBR 64k (-vbr off) to VBR 48k with -application voip and
-compression_level 10. The old CBR settings produced lower-quality speech
audio and occasionally failed to trigger Telegram's native voice-bubble
rendering. Applied to _ffmpeg_transcode_to_opus (the single transcoder
behind _convert_to_opus and _repair_ogg_container), the Gemini WAV→OGG
path, and the Matrix adapter boundary transcoder added in this branch.
Fixes#18818
Salvaged from PR #68063 (@malaiwah). MatrixAdapter.send_voice now transcodes
any non-Ogg audio to Ogg/Opus at the adapter boundary (best-effort — the
original file is sent unchanged when ffmpeg is unavailable), so MSC3245
voice bubbles render even when a caller hands the adapter MP3/WAV audio.
_matrix_voice_metadata_for_file probing now runs via asyncio.to_thread so
ffprobe/ffmpeg subprocess timeouts can't stall the adapter event loop.
The PR's tools/tts_tool.py want_opus hunk was dropped: main's
OPUS_VOICE_PLATFORMS set (PR #73072) already includes matrix; the Matrix
opus-routing test is kept.
Refs #14841
Salvaged from PR #68063 (base commit, runner-path hunks superseded by the
platform-aware OPUS_VOICE_PLATFORMS fix in this branch). Element and other
Matrix clients render voice bubbles more reliably when m.audio events carry
duration and waveform metadata; probe both best-effort via ffprobe/ffmpeg.
Consolidates all TTS text-preparation paths onto
tools/tts_text_normalize.prepare_spoken_text:
- strip_nonspoken_blocks: removes <think> reasoning blocks (#34213,
incl. unterminated streaming blocks) and the end-of-turn
file-mutation verifier footer emitted by run_agent.py (#40772).
- flatten_newlines_for_payload: collapses newlines into sentence
breaks so newline-sensitive OpenAI-compatible providers (Kokoro)
speak the whole script instead of truncating at the first newline
(#9004).
- tools/tts_tool._strip_markdown_for_tts (voice-mode streaming + web
dashboard path) now delegates to the shared cleaner, with the legacy
regex pipeline kept as a best-effort fallback.
- hermes_cli/voice.py speak_text and cli.py _voice_speak_response now
use the shared cleaner instead of their own duplicated regex
pipelines.
- gateway auto-TTS fallback also strips think blocks.
Tests: tests/tools/test_tts_prepare_spoken.py covers think blocks,
verifier footer, emoji, newline flattening, and the shared-cleaner
wiring on the tool/streaming/gateway paths. Updated the header
expectation in test_voice_cli_integration.py for the heading-fold
behavior of the shared cleaner.
Closes#34213, #9004, #40772
Address review feedback on #31693:
- Regression tests assert extra_body == {"lang_code": ...} is forwarded
when tts.openai.language is configured, and omitted when unset/empty
- Document tts.openai.language as intended for OpenAI-compatible
endpoints that support lang_code (e.g. Kokoro-FastAPI)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 'language' config field in tts.openai.language was read but never passed
to the API. This caused Kokoro (and other OpenAI-compatible TTS backends that
support lang_code) to default to English phonemization regardless of the
configured language.
Now passes lang_code via extra_body when language is set in config.
xAI TTS supports a text_normalization boolean that normalizes
written-form text (numbers, abbreviations, symbols) into spoken-form
before synthesis. This parameter was not being sent, leaving users
with literal number/symbol pronunciation.
- Add DEFAULT_XAI_TEXT_NORMALIZATION_DEFAULT = False
- Read tts.xai.text_normalization from config (via _xai_bool_config)
- Attach text_normalization: true to the POST payload when enabled
- Omit the field entirely when unset or explicitly false (API default)
- Add 3 tests: default omission, enabled passthrough, explicit false
Adds an optional provider parameter to the text_to_speech tool that lets the model select a TTS provider per-call instead of always using the globally configured tts.provider.
When provider is set, it bypasses the configured default and routes directly to the specified backend.
When omitted (the default), the tool behaves exactly as before.
Closes#47459
The `text_to_speech` tool schema accepted only `text` and `output_path`,
so style direction (tone, emotion, pacing, whispering) could never reach
the OpenAI backend — even though `gpt-4o-mini-tts` (Hermes's OpenAI
provider default) treats `instructions` as its primary voice-design
control.
This plumbs an optional `instructions` argument through the tool schema,
the handler lambda, and `text_to_speech_tool()` into
`_generate_openai_tts`, where it is forwarded to
`client.audio.speech.create()` only when truthy. Empty/None values still
omit the key entirely, preserving behavior on `tts-1`/`tts-1-hd` and
strict OpenAI-compatible servers.
The same passthrough unblocks self-hosted OpenAI-compatible voice-design
servers (Qwen3-TTS-VoiceDesign on oMLX, etc.) that are already wired in
via `tts.openai.base_url` — the established convention per #9004 and the
TTS config docs — without inventing a new provider backend.
Tests: `tests/tools/test_tts_instructions.py` covers backend passthrough,
tool-level threading, schema declaration, and the empty-string/absent
omission cases. `tests/tools/test_tts_max_text_length.py` fake_openai
signature widened to accept the new kwarg.
Refs NousResearch/hermes-agent#14196
Add optional 'speed' parameter (0.25-4.0) to the text_to_speech tool
schema and handler. When provided by the model, it overrides the
config-level tts.speed setting, enabling per-request speed control
without config changes.
Use cases:
- Language learning: slow playback (0.5x) for pronunciation practice
- Accessibility: adjustable speed for hearing preferences
- Content review: accelerated playback for long text
The speed value is clamped to [0.25, 4.0] and injected into tts_config
before dispatching to any provider (Edge, OpenAI, MiniMax, etc.), so
all existing provider-level speed handling works transparently.
Includes 3 new tests for tool-level speed injection, clamping, and
config preservation when speed is not specified.
When /reasoning show is enabled, the model's <think> blocks appear in
the final assistant message. TTS reads these aloud, which is unwanted —
users want to see reasoning but not hear it spoken.
- Add _THINK_BLOCK regex to _strip_markdown_for_tts() (tools/tts_tool.py)
— the general TTS text-preparation path used by all TTS providers
- Add <think> stripping to prepare_tts_text() (gateway/platforms/base.py)
— the gateway auto-TTS path
- Reuses the existing regex pattern from stream_tts_to_speaker()
Closes#34213
Review follow-up: the spoken script is for synthesis only. Caption
eligibility and payload stay on the original reply, so a long reply
whose normalized script fits the 1024 char limit is still delivered
in full as its own message. Adds the long-original/short-normalized
regression case to the auto-TTS caption tests.
Auto-TTS previously fed raw chat Markdown and compact symbols straight to the
speech provider, so units were read as stray letters and headings or bullets ran
together. This routes spoken text through a new normalizer,
tools/tts_text_normalize.prepare_spoken_text, that expands units (for example a
temperature written with the degree symbol becomes "degrees Celsius") and
flattens Markdown into a transcript-like script with sentence pauses.
The normalizer is best-effort: if it ever fails the code falls back to the
previous markdown-strip behavior, so auto-TTS keeps working. The Telegram voice
caption uses the same normalized text. Includes a unit test.
Follow-up to the #60063 salvage: the curated gemini list now carries
gemini-3.6-flash (aux default, #70416) and the vertex list carries
gemini-3.5-flash-lite (#68767) — both need snapshot pricing so direct
Gemini/Vertex sessions don't report cost=unknown.
Rates verified against https://ai.google.dev/gemini-api/docs/pricing
(2026-07-28): 3.6-flash $1.50/$7.50, cache read $0.15;
3.5-flash-lite $0.30/$2.50, cache read $0.03.
Main gained its own vertex curated list (df051c17cc) two days after
PR #68767 was opened, so the cherry-pick produced a duplicate 'vertex'
dict key (later key silently wins in Python dict literals). Merge the
two into one list: union of both, existing entries preserved, contributor's
live-validated additions (gemini-3.6-flash, 3.5-flash-lite, 3.1-flash-lite)
folded in.
Add a "vertex" key to _PROVIDER_MODELS with 6 Gemini 3.x models that
were validated live against the Vertex AI OpenAI-compatible endpoint
(aiplatform.googleapis.com, global region, project antse-tooling) on
2026-07-21. All entries returned HTTP 200; 8 other candidates (e.g.
gemini-3.1-flash, gemini-3-pro-preview) returned 404 and were excluded.
Validated models (google/ prefix required by Vertex endpoint):
- google/gemini-3.5-flash (frontier Flash, agentic)
- google/gemini-3.6-flash (newer incremental over 3.5)
- google/gemini-3.5-flash-lite (lighter/cheaper 3.5 variant)
- google/gemini-3.1-pro-preview (3.1 Pro preview)
- google/gemini-3-flash-preview (3.0 Flash preview)
- google/gemini-3.1-flash-lite (most cost-efficient 3.x model)
Context lengths are covered by the existing "gemini" prefix entry
(1_048_576) in agent/model_metadata.py DEFAULT_CONTEXT_LENGTHS — no
additional entries needed.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The native Gemini provider profile's default_aux_model and the curated
model picker catalog were still pinned to gemini-3.5-flash, a stale
generation now superseded by gemini-3.6-flash (documented GA). Bump
both so the auxiliary-task default and the picker stay in sync with
the current model.
Contract test asserts the durable lockstep invariant only
(default_aux_model is a member of _PROVIDER_MODELS["gemini"]) rather
than pinning either side to a frozen model-name string, so it doesn't
need updating on the next model-generation bump.
Salvaged from PR #62977 (@Vissirexa) — TTS model-cache half only (the
hindsight turn-buffer half is a different subsystem and was dropped).
_piper_voice_cache and _kittentts_model_cache were keyed by voice/model
with no eviction, and each entry is a whole loaded model (tens of MB).
A surface that sweeps voices pinned one model per voice for the process
lifetime. New _tts_cache_get_or_load() get-or-loads through a small LRU
(_TTS_MODEL_CACHE_MAX=3), refreshing recency on a hit and evicting the
least-recently-used model on a cold miss.
Class-level sweep following the ElevenLabs salvage (#66311): every cloud
TTS provider section now honors tts.<provider>.base_url. xAI, MiniMax,
Gemini, OpenAI and DeepInfra already did; Mistral (SDK server_url) was
the remaining gap. Adds per-provider config tests locking in the
ElevenLabs environment plumbing and the Mistral server_url passthrough.
Salvaged from PR #66311 (@moeadham), rebased onto the current streaming
registry. tts.elevenlabs.base_url (+ optional wss_url, derived from
base_url when omitted) routes both the sync ElevenLabs path and the
chunked ElevenLabsStreamer through an ElevenLabsEnvironment, matching
the STT side's ELEVENLABS_STT_BASE_URL/config override pattern.
Follow-up to salvaged PR #70307 (@aml1973): OpenAIStreamer now checks
tts.openai.api_key (config.yaml) ahead of the env resolver in both
available() and stream(), completing config parity between the sync
and streaming OpenAI TTS paths.
Use the shared OpenAI audio key resolver and prefer tts.openai.base_url over the global environment fallback in the Desktop streaming path. Add focused regression coverage for credential and endpoint propagation.
