Replaces the half-duplex per-playback barge monitors with ONE listener
that runs for the entire agent turn in continuous voice mode: armed at
utterance-submit, disarmed when the turn is fully done (response + TTS
finished). Fixes Teknium's live report that voice interruption never
works: (a) not while the LLM is generating, (b) not while TTS plays.
Root causes:
- HALF-DUPLEX GAP: the barge monitor only spawned when TTS playback
STARTED (cli.py streaming/whole-file paths, gateway _tts_stream_begin).
During LLM generation there was NO microphone listener at all.
- PLAYBACK DEAFNESS: the monitor calibrated its VAD noise floor WHILE the
speaker was blasting TTS (speaker bleed baked into the floor), then
multiplied it by 8x with a 1s strictly-consecutive block requirement —
normal speech could rarely reach the trigger, and the 2s grace
swallowed early interjections.
New model — tools/voice_mode.full_duplex_listen():
- Pre-playback calibration: quiet-room noise floor established at turn
start and HELD through playback (never recalibrated against bleed).
- Phase-aware trigger: generation = floor x voice.barge_in_threshold_multiplier
(new config, default 3.0, justified by synthetic-frame tests);
playback = additionally clamped to a 1500-RMS minimum so bleed alone
can't trip; 4000-RMS ceiling keeps speech always reachable.
- Windowed-majority detection (>=80% of a 300ms window) instead of the
strictly-consecutive counter that reset on intra-word energy dips.
- Grace on playback ONSET only (voice.barge_in_grace_seconds, default
down 2.0 -> 0.5) — suppresses the onset transient, not the mic.
- Debug diagnostics at every decision point (calibrated floor, per-window
RMS above 50% of trigger, trip/no-trip, grace suppressions) — always
logger.debug, mirrored to stderr under HERMES_VOICE_DEBUG=1.
Phase behavior (CLI cli.py + tui_gateway/server.py, same model):
- generation: speech interrupts the in-flight turn via the SAME seam the
typed/Ctrl+C interrupt uses (agent.interrupt()), cuts any pending TTS
pipeline so the stale reply never plays, and submits the captured
interjection (pre-roll capture, first syllable kept) as the next turn.
- playback: cuts TTS (streaming pipeline stop + fallback speak stop
events + file player) and submits the capture.
- stop phrase honored in BOTH phases: mid-generation 'stop' interrupts
the turn AND ends the voice chat (stop everything).
- one listener instance spans generation -> playback (no re-arm race);
double-arm refused (CLI _voice_fd_active / gateway _fd_listener_active).
Gateway specifics: _arm_full_duplex_listener() at _run_prompt_submit turn
start and inside _tts_stream_begin; _speak_text_with_barge registers its
(stop, done) pair in _fd_speak_pipelines so fallback speaks are cut and
tracked; _tts_stream_barge_in_monitor kept as a shim that arms the new
listener. Desktop renderer owns its own mic path (voice-barge-in.ts) and
is unaffected; if desktop backend-mic mode is used it inherits via the
gateway.
Tests: full_duplex_listen synthetic-RMS suite (speech-over-bleed trips,
bleed alone doesn't, quiet floor held through playback, grace window,
multiplier math 3x vs 8x, windowed-majority dips), CLI listener phase
tests (generation interrupt seam, playback cut, lifecycle spans phases,
double-arm, config forwarding, stop-phrase-mid-generation), gateway
generation-phase interrupt + stop-phrase tests. Generation-interrupt
test sabotage-verified.
Saying OR typing a configured stop phrase (voice.stop_phrases, default
"stop") now ends the voice chat everywhere, not just classic CLI PTT:
- hermes_cli/voice.py: new explicit on_stop_phrase callback through
start_continuous/stop_continuous. The force-transcribe path previously
DISCARDED the stop phrase silently — with auto_restart=False the client
re-arms the next capture, so the conversation never ended. Both halt
paths now fire on_stop_phrase (fallback: on_silent_limit for legacy
callers) as user intent, distinct from the no-speech timeout.
- tui_gateway/server.py: voice.record wires on_stop_phrase and emits
voice.transcript {stop_phrase: true} after flipping HERMES_VOICE(_TTS)
off and stopping streaming TTS — same teardown as /voice off. The TTS
barge-in monitor stop-checks its transcript too. prompt.submit consumes
a TYPED bare stop phrase at the server-side choke point when voice mode
is on (returns {voice_stopped: true}, no turn starts).
- ui-tui: voice.transcript {stop_phrase} ends voice mode with a clear
'voice chat ended' notice (distinct from the no-speech-limit message);
submitPrompt releases the busy latch on a consumed voice_stopped reply.
- cli.py: _typed_voice_stop in process_loop — typing a bare stop phrase
while voice mode/continuous is active ends voice mode instead of
sending 'stop' to the agent; typed 'stop' outside voice mode is
unchanged. Voice transcripts skip the check (already stop-checked).
- desktop: interceptsTypedVoiceStop — the composer's onSubmit ends the
live voice conversation (same path as clicking end on the pill) when a
bare stop command is typed with no attachments; renderer-owned loop, so
handled client-side like the existing spoken isVoiceStopCommand.
- tools/voice_mode.py: transcribe_recording never lets the Whisper
hallucination filter swallow a configured stop phrase (e.g. 'bye'
configured as a stop phrase is both a hallucination-blocklist entry and
a stop phrase — stop-phrase check now wins).
Tests: continuous-loop signal (sabotage-verified), force-transcribe stop
signal + legacy fallback, hallucination-filter ordering, typed-stop CLI
unit tests (voice on/off/longer text), prompt.submit typed-stop gateway
tests, TUI vitest for stop_phrase event handling, desktop vitest for the
typed-stop interceptor.
Adds gpt-transcribe (OpenAI's new file-transcription model, $0.0045/min)
to the OpenAI STT provider:
- OPENAI_MODELS set: gpt-transcribe is recognized so provider
auto-correction keeps it on OpenAI and rejects it on Groq
- Language hint wiring: gpt-transcribe replaces the singular
'language' field with a 'languages' list; the API rejects the legacy
field, so the hint is sent via extra_body {languages: [..]}
- Config comment (DEFAULT_CONFIG), cli-config.yaml.example, desktop
settings enum, and docs (en + zh-Hans) updated
- Tests: model pass-through, languages-list hint shape, legacy singular
hint preserved for gpt-4o-transcribe, Groq auto-correction
gpt-live-transcribe (realtime WebSocket, $0.017/min) is NOT wired here:
the file-based STT pipeline has no realtime session path; it belongs in
a future realtime/voice-mode integration.
The `text_to_speech` tool schema accepted only `text` and `output_path`,
so style direction (tone, emotion, pacing, whispering) could never reach
the OpenAI backend — even though `gpt-4o-mini-tts` (Hermes's OpenAI
provider default) treats `instructions` as its primary voice-design
control.
This plumbs an optional `instructions` argument through the tool schema,
the handler lambda, and `text_to_speech_tool()` into
`_generate_openai_tts`, where it is forwarded to
`client.audio.speech.create()` only when truthy. Empty/None values still
omit the key entirely, preserving behavior on `tts-1`/`tts-1-hd` and
strict OpenAI-compatible servers.
The same passthrough unblocks self-hosted OpenAI-compatible voice-design
servers (Qwen3-TTS-VoiceDesign on oMLX, etc.) that are already wired in
via `tts.openai.base_url` — the established convention per #9004 and the
TTS config docs — without inventing a new provider backend.
Tests: `tests/tools/test_tts_instructions.py` covers backend passthrough,
tool-level threading, schema declaration, and the empty-string/absent
omission cases. `tests/tools/test_tts_max_text_length.py` fake_openai
signature widened to accept the new kwarg.
Refs NousResearch/hermes-agent#14196
Saying EXACTLY a configured stop phrase (default: 'stop') and nothing
else now ends the voice conversation instead of being sent to the agent
as a prompt. Match is deliberately strict — whole utterance,
case-insensitive, surrounding punctuation stripped — so 'stop doing
that and try X' still reaches the agent.
- tools/voice_mode.py: is_voice_stop_phrase() + voice.stop_phrases
config loader (default ['stop'], [] disables, malformed config falls
back safely).
- hermes_cli/voice.py: shared continuous loop (TUI + desktop) halts on
a stop phrase exactly like the silent-cycle limit (fires
on_silent_limit so every UI turns voice off); stop_continuous
force-transcribe path swallows the phrase without counting a silent
cycle.
- cli.py: classic CLI push-to-talk/continuous path and barge-in
utterance path disable voice mode on a stop phrase.
- Config default voice.stop_phrases: ['stop']; docs updated.
Tests: tests/tools/test_voice_stop_phrase.py (27 tests,
sabotage-verified: loop test fails when detection is disabled);
voice suites green (110 + 44 passed).
- Normalize stt.groq.language: cast to str, strip, treat
empty/whitespace as unset (parity with xAI's str().strip()).
- Clarify "blank = auto-detect" inline comments in 4 docs/configs to
reflect the env-var fallback (HERMES_LOCAL_STT_LANGUAGE).
- Document that HERMES_LOCAL_STT_LANGUAGE also drives the local
faster-whisper provider, not just the CLI fallback.
- Add unit tests covering: omitted language when unset, config-supplied
language, env fallback, config-over-env precedence, whitespace
normalized to unset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read stt.groq.language from config.yaml (with HERMES_LOCAL_STT_LANGUAGE
env fallback) and forward it to the Groq Whisper API to skip
auto-detection on known-language audio. Omit when unset so Groq
auto-detects, preserving today's behavior. Bonus: swap xAI's hardcoded
env literal for the LOCAL_STT_LANGUAGE_ENV constant for consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
markVoicePlaybackInterrupted() / takeVoicePlaybackInterrupted() mirror
the backend latch in the renderer (the barge happens client-side, where
the audio plays). VAD barges and typing over playback mark it; the next
prompt.submit carries interrupted:true, which the TUI gateway latches
into the model note.
* docs: stop recommending pip install hermes-agent; point to install script
The install script is the only supported install path (it provisions a
managed, isolated uv environment). Replace bare `pip install hermes-agent`
primary-install recommendations with the curl install script, and rewrite
optional-extra snippets (`pip install "hermes-agent[X]"`) to the managed-env
form `cd ~/.hermes/hermes-agent && uv pip install -e ".[X]"` that matches the
installer and the English quickstart.
Covers English docs + zh-Hans mirrors, the achievements plugin README, and
realigns the zh-Hans quickstart to the English Desktop-installer-first layout
(dropping its stale "Method A — pip (simplest)" section).
* docs: drop pip as a supported install/update method
Removes the 'pip installs' supported-method sections from updating.md and
cli-commands.md (EN + zh-Hans): the curl install script is the only supported
way to install/update the Hermes CLI. The _cmd_update_pip pip/pipx branches
remain in code as an undocumented safety net for users who already have such an
install, but the docs no longer advertise pip as a path.
Also normalizes a bare `pip install -e '.[acp]'` to the managed-env form.
Leaves python-library.md untouched: importing AIAgent as a library dependency
into your own project is a distinct use case where pip is correct.
The locale switcher appeared broken because hardcoded markdown links
(`](/docs/X)`) got double-prefixed by Docusaurus to `/docs/<locale>/docs/X`
(404) in non-English locales, and the MDX hero `<a href>` on the index
page escaped locale routing entirely.
Changes:
- Rewrite 922 `](/docs/X)` -> `](/X)` across 166 docs files (strip trailing
.md too). Docusaurus prepends locale + baseUrl itself.
- docs/index.md -> index.mdx; hero "Get Started" anchor -> Docusaurus
<Link> so it stays inside the active locale.
- Drop `ko` locale entirely from docusaurus.config.ts + delete i18n/ko/
(4 stale auto-translated kanban pages, <2% coverage, misleading).
Verified `npm run build` succeeds for both en and zh-Hans; `build/zh-Hans/
index.html` has no /docs/zh-Hans/docs/... double-prefixed paths.
PR2 will translate the 335 English docs into i18n/zh-Hans/.
Follow-up to #30869. Adds Portal mentions on user-facing pages that
naturally call for an LLM + tool credentials but didn't previously
acknowledge Portal as a one-stop option.
- getting-started/installation.md: tip after the 'after install' block
pointing at 'hermes setup --portal' for users who want everything wired
at once instead of piecewise via 'hermes model' + 'hermes tools'.
- user-guide/configuring-models.md: small tip near the top — the page is
literally about provider/model choice and previously had zero Portal
mention.
- user-guide/features/voice-mode.md: Prerequisites need both an LLM and
TTS — a Portal subscription is the single setup that covers both.
- user-guide/features/batch-processing.md: highlights Portal as a
predictable-cost option for parallel agent runs that hit many APIs.
- user-guide/features/api-server.md: backend needs models + tools; one
Portal sub gives a fully-equipped OpenAI-compatible endpoint.
- user-guide/windows-native.md: early-beta users on Windows benefit most
from skipping per-tool Windows-key-juggling.
- integrations/providers.md: updates the existing Tool Gateway tip and
the Nous Portal section to mention the new commands.
- user-guide/features/fallback-providers.md: Nous row in the provider
table now lists 'hermes setup --portal' as the fresh-install path.
Tone discipline: one Portal mention per page, concrete CLI commands
(no marketing copy), always solving a problem the page itself sets up.
Two coordinated changes that unblock downstream audio pipelines
(diarization, custom transcription, archival) on attachments larger
than the public Bot API's 20MB getFile ceiling.
- `stt.enabled: false` no longer drops voice/audio with a generic
"transcription disabled" note. The gateway probes the cached file's
duration (wave → mutagen → ffprobe ladder) and surfaces
`[The user sent a voice message: <abs path> (duration: M:SS)]` to
the agent so a skill or tool can pick up the raw file. The previous
placeholder is replaced rather than appended when present.
- `platforms.telegram.extra.base_url` set → adapter auto-lifts its
document size cap from 20MB to 2GB (the local telegram-bot-api
`--local` ceiling) and the "too large" reply reports the active
limit dynamically. No new config knob; presence of `base_url` is the
opt-in.
- `platforms.telegram.extra.local_mode: true` wires
`Application.builder().local_mode(True)` on the python-telegram-bot
builder. PTB then reads files from disk instead of HTTP, which is
required when telegram-bot-api runs in `--local` mode (the server
returns absolute filesystem paths, not `/file/bot...` URLs).
- gateway/run.py: rewrites the `stt.enabled: false` branch of
`_enrich_message_with_transcription`. New `_format_duration` +
`_probe_audio_duration` helpers.
- gateway/platforms/telegram.py: `_max_doc_bytes` instance attribute
derived from `extra.base_url`; `local_mode` builder wiring;
dynamic "too large" message.
- tests/gateway/test_stt_config.py: covers path-surfacing with and
without an existing user message, and placeholder replacement.
- tests/gateway/test_telegram_max_doc_bytes.py: 3 cases — default 20MB
without base_url, 2GB when set, empty-string base_url keeps default.
- website/docs/user-guide/messaging/telegram.md: new "Skipping STT"
subsection under Voice Messages and a full "Large Files (>20MB) via
Local Bot API Server" walkthrough (api_id/api_hash, docker-compose,
one-time `logOut` migration, `platforms.telegram.extra` config, the
`local_mode` disk-access requirement, the silent HTTP-fallback 404).
- website/docs/user-guide/features/voice-mode.md: documents the
`stt.enabled` knob in the config reference.
- `pytest tests/gateway/test_telegram_max_doc_bytes.py
tests/gateway/test_stt_config.py` → 9/9 passing.
- Verified end-to-end on a live deployment: gateway log shows
`Using custom Telegram base_url: http://...` and
`Using Telegram local_mode (read files from disk)` on startup;
voice messages above 20MB cache to disk and surface their path to
the agent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Salvage of #11350. Kept:
- Code: add an explicit /voice join Choice in the slash UI (runner accepts both 'join' and 'channel' but only 'channel' was in autocomplete).
- Docs: Server Members Intent is conditional (only needed if DISCORD_ALLOWED_USERS contains usernames); SSRC → user_id mapping uses the voice websocket SPEAKING opcode, not the Members intent.
Dropped from the original PR:
- HERMES_DISCORD_VOICE_PACKET_DUMP — this env var doesn't exist on main (it was in a different PR that isn't merged).
- DISCORD_PROXY docs — already documented on current main.
- DISCORD_ALLOW_MENTION_* docs — already on main.
- "barge-in mode" rewrite — current main actually does pause the listener during TTS (VoiceReceiver.pause() at discord.py:192); there is no barge_in_guard/barge_in_rms on main.
Co-authored-by: Michel Belleau <michel.belleau@malaiwah.com>
zsh interprets square brackets as glob patterns, so
`pip install hermes-agent[voice]` fails with 'no matches found'.
Quote all pip install commands with extras across 5 docs pages (12 instances).
Reported by OFumik0OP.
Allow users to configure a custom base_url for the OpenAI TTS provider
in ~/.hermes/config.yaml under tts.openai.base_url. Defaults to the
official OpenAI endpoint. Enables use of self-hosted or OpenAI-compatible
TTS services (e.g. http://localhost:8000/v1).
Also adds a TTS configuration example block to cli-config.yaml.example.
Add a hands-on guide for using voice mode with Hermes, fix and expand the main voice-mode docs, surface /voice in messaging docs, and improve discoverability from the homepage and learning path.
Remove web UI gateway (web.py, tests, docs, toolset, env vars, Platform.WEB
enum) per maintainer request — Nous is building their own official chat UI.
Fix 1: Replace sd.wait() with polling pattern in play_audio_file() to prevent
indefinite hang when audio device stalls (consistent with play_beep()).
Fix 2: Use importlib.util.find_spec() for faster_whisper/openai availability
checks instead of module-level imports that trigger heavy native library
loading (CUDA/cuDNN) at import time.
Fix 3: Remove inspect.signature() hack in _send_voice_reply() — add **kwargs
to Telegram send_voice() so all adapters accept metadata uniformly.
Fix 4: Make session loading resilient to removed platform enum values — skip
entries with unknown platforms instead of crashing the entire gateway.
Merge main's faster-whisper (local, free) with our Groq support into a
unified three-provider STT pipeline: local > groq > openai.
Provider priority ensures free options are tried first. Each provider
has its own transcriber function with model auto-correction, env-
overridable endpoints, and proper error handling.
74 tests cover the full provider matrix, fallback chains, model
correction, config loading, validation edge cases, and dispatch.
Code fixes:
- STT model, Groq base URL, and OpenAI STT base URL are now
configurable via env vars (STT_GROQ_MODEL, STT_OPENAI_MODEL,
GROQ_BASE_URL, STT_OPENAI_BASE_URL) instead of hardcoded
- Gateway and Discord VC now read stt.model from config.yaml
(previously only CLI did this — gateway always used defaults)
Doc fixes:
- voice-mode.md: move Web UI troubleshooting to web.md (was duplicated)
- voice-mode.md: simplify "How It Works" for end users (remove NaCl,
DAVE, RTP internals)
- voice-mode.md: clarify STT priority (OpenAI used first if both keys
set, Groq recommended for free tier)
- voice-mode.md: document new STT env overrides in config reference
- web.md: remove duplicate Quick Start / Step 1-3 sections
- web.md: add mobile HTTPS mic workarounds (moved from voice-mode.md)
- web.md: clarify STT fallback order
- voice-mode.md: Discord sends native voice bubbles (OGG/Opus flags=8192),
not MP3 file attachments. Falls back to file only if voice API fails.
- discord.md: Bot requires @mention by default in server channels
(DISCORD_REQUIRE_MENTION=true). Previous text incorrectly said no
mention needed.
- index.md: Fix broken ASCII architecture diagram alignment after
adding Web adapter box.
- macOS firewall may block LAN access to Web UI
- Mobile browsers require HTTPS for microphone API
- Document workarounds: Android Chrome flag, mkcert self-signed cert,
Caddy reverse proxy, SSH tunnel for iOS
- Document DM vs server channel interaction modes
- Explain @mention requirement and how to select bot user vs role
- Add DISCORD_REQUIRE_MENTION and DISCORD_FREE_RESPONSE_CHANNELS config
- Add troubleshooting entry for bot not responding in server channels
Cover CLI voice mode, Telegram/Discord auto voice reply, and Discord
voice channel support. Include setup guide with bot permissions, OAuth2
invite URL, privileged intents, system dependencies, and Python packages.
Update discord.md voice messages section with correct STT key reference.
