The _is_voice_content_type() heuristic matched audio file extensions
(.wav, .mp3, .ogg, etc.) even when the QQ Bot API explicitly reported
content_type='file'. This caused files sent via QQ's file-transfer
feature to be routed through the speech-to-text pipeline instead of
being saved as regular attachments.
The QQ Bot API already distinguishes voice messages (content_type=
'voice') from file uploads (content_type='file'), so filename-based
extension sniffing is unnecessary and harmful.
Removed the _VOICE_EXTENSIONS fallback; now only content_type is
checked.
Closes #XXXX
The msg_type_str == "richText" branch reset msg_type to PHOTO/TEXT after
the rich-text item scan had already promoted it (e.g. a native voice item
→ VOICE), dropping voice notes from the auto-STT path. Only re-derive when
the scan left the type at TEXT.
Ports the root-cause analysis from PR #38276 (stale, targeted the deleted
gateway/platforms/dingtalk.py) onto the live plugin adapter, with
regression tests.
Refs #38211#38219#38276
- Promote EXT_MAP to a module-level constant for reuse
- Classify DingTalk image messages and image file attachments as PHOTO
- Extract DingTalk card/interactiveCard document link content with defensive parsing
- Handle None, empty, JSON, and plain-string card content fields
- Add coverage: 8 card + 4 interactiveCard + 5 _extract_media = 17 test cases
- Remove a shadowed duplicate TestExtractText class (pytest collected only the later one)
3 fixes for the dingtalk-platform plugin (plugins/platforms/dingtalk/adapter.py):
1. _extract_text: parse ASR recognition text from voice messages via
extensions['content']['recognition'], fall back to fileName for
file messages ("[文件] xxx")
2. _extract_media: handle audio/file/image msgtype_str with correct
MIME mapping; exclude audio from media_urls to preserve DingTalk's
built-in ASR result (avoids failed whisper re-transcription)
3. _resolve_media_codes: parse downloadCode from extensions['content']
for file/image message types
All tested with live DingTalk group chat: voice → recognition text,
PDF/Markdown file → filename + download.
Trimmed cherry-pick of PR #40592 (duration + dedup hunks only; the
voice-classification hunk duplicates #29235 and the send_voice Opus
rewrite is out of scope for this inbound-focused PR):
- adapter.py: ffprobe duration (off-loop) attached to Feishu voice
uploads via _build_file_upload_body(duration=...) and the audio
message payload (#16524, #8300)
- gateway/run.py: TTS dedup narrowed to the current turn;
_enrich_message_with_transcription dedups repeated audio paths
Refs #40592#16524#8300
Lark's native "audio" msg_type is an in-app voice recording — uploaded
audio files arrive as "file"/"media". But _resolve_normalized_message_type
resolved the "audio" preferred type to MessageType.AUDIO, which the gateway
treats as a non-transcribed file attachment (run.py: AUDIO -> audio_file_paths,
"never STT"; VOICE -> audio_paths, "always STT"). Result: a Feishu voice
note reached the agent as an untranscribable audio attachment and was
silently ignored — the user's spoken message never became text.
Every other platform that receives native voice notes (Telegram, Discord,
Slack, WhatsApp, Signal, Matrix, WeChat, WeCom, DingTalk, QQ, BlueBubbles,
Mattermost, Yuanbao) classifies them as MessageType.VOICE. Feishu was the
only one classifying them as AUDIO. This is the follow-up to #28993, which
added native voice-note transcription for Discord + DingTalk but did not
cover Feishu.
Return MessageType.VOICE for the "audio" branch. The branch is reached only
for Lark's top-level audio msg_type (set in the normalizer; file uploads map
to "document"), so VOICE is unconditionally correct here — no risk of
auto-transcribing an uploaded music/audio file.
- plugins/platforms/feishu/adapter.py: _resolve_normalized_message_type audio
branch returns VOICE instead of resolving to AUDIO via mime.
- tests/gateway/test_feishu.py: test_extract_audio_message_downloads_and_caches
asserted the old AUDIO behavior on a fixture literally named voice.ogg —
updated to expect VOICE (the corrected classification).
- tests/gateway/test_feishu_voice_message_type.py: new focused regression
tests (audio->VOICE with and without mime; photo/document/text unaffected).
Rebased onto current main: the Feishu platform moved from the single-file
gateway/platforms/feishu.py into the plugins/platforms/feishu/ package; the
fix applies to the same _resolve_normalized_message_type logic at its new home.
Verified the new voice tests fail when the branch resolves to AUDIO and pass
with VOICE, while the photo/document/text cases are unaffected either way.
Note: classification is mock-tested here; the downstream STT pipeline is the
shared, already-proven path (#28993). End-to-end verification on a live
Feishu account would be a welcome confirmation.
Refs #28993 (sibling-gap: Feishu was the platform left uncovered).
Salvage of PR #52188. The original PR raised RuntimeError when LM Studio
load was rejected or unverifiable, which would abort agent startup on
transient network failures. Replace with logger.warning + fallback to
configured context length, preserving the old graceful-degradation behavior.
Bare-login noreply emails (no NNN+ numeric prefix) do not match the
check-attribution auto-resolve rule, so they need an explicit
contributors/emails/ file. Unblocks PR #73592.
* fix(desktop): allow camera capture through the permission handlers
The session permission hooks were written for the voice composer and denied
video outright, so any renderer getUserMedia({video}) failed with
NotAllowedError before the OS was ever consulted.
Rename isAudioCapturePermission to isMediaCapturePermission and accept video
alongside audio in both the request and check handlers. The OS capture
permission still applies, so the user keeps a real allow/deny.
* build(desktop): declare camera usage for signed macOS builds
A hardened-runtime build needs the camera entitlement and an
NSCameraUsageDescription string, or the packaged app is killed on first
camera access instead of prompting.
_generate_xai_tts passes stream=True to requests.post since the TTS
speaker pipeline moved to the streaming core. Four fake_post mocks in
test_tts_xai_speech_tags.py still had the pre-streaming signature and
raised TypeError: unexpected keyword argument 'stream'. The other 20+
mocks in the same file already accept stream=False; this aligns the
stragglers.
Fixes the 4 failures in Python tests slice 5/8 on main.
The codex app-server namespaces MCP tool call ids as
codex_mcp__<server>__<tool>_<codex_call_id>. With an exec-<uuid> component the
built-in hermes-tools server alone overflows the Responses API's 64-char
call_id limit, so the request 400s with a non-retryable "string too long".
The offending item sits near the head of the transcript and replays every
turn, permanently bricking the session — the only recovery is /reset.
Sibling defect to #10788, which clamped input[*].id via
_MAX_RESPONSES_ITEM_ID_LENGTH. Apply the same treatment to call_id at both
Responses emit sites in _chat_messages_to_responses_input: a deterministic
surrogate (call_ + sha256[:32]) for ids over the limit, short ids unchanged.
Because the surrogate is a pure function of the original id, a function_call
and its matching function_call_output — which carry the same original id — map
to the same surrogate and stay paired without correlating the two items.
Two log-spam bugs found in live gateway logs:
1. gateway_routing UNIQUE-constraint spam (261 warnings in one errors.log):
early builds of the #59203 routing-index migration created
gateway_routing with 'session_key TEXT PRIMARY KEY' and no scope
column. _reconcile_columns() ADDs the missing scope column but SQLite
cannot ALTER a primary key, so the shipped composite
PRIMARY KEY (scope, session_key) never lands on those databases. Both
write paths then fail on every save:
- save_gateway_routing_entry: 'ON CONFLICT clause does not match any
PRIMARY KEY or UNIQUE constraint'
- replace_gateway_routing_entries: 'UNIQUE constraint failed:
gateway_routing.session_key' whenever the same session_key exists
under another scope (e.g. test-suite scopes leaked into a live DB).
New _heal_gateway_routing_pk() rebuilds the table once with the
composite key, preserving rows (newest wins on collisions, NULL scope
coalesced to ''). Same one-time-heal pattern as the #51646 active-
column repair. Verified E2E against a copy of a real affected state.db.
2. pre_gateway_dispatch warned ''GatewayRunner' object has no attribute
'session_store'' and silently dropped the hook for every message on
partially-initialized runners (bare object.__new__ runners in tests,
and any future init-order change). Pass
getattr(self, 'session_store', None) so the hook always fires
(pitfall #17 pattern).
Both regression tests fail without their fixes (sabotage-verified).
The /api/audio/speak-stream WebSocket URL was always minted for the primary
backend with no profile param, so streaming read-aloud used the default
profile's TTS provider even when another profile was active. Mint the
ticket for the active profile's connection and carry ?profile= (same seam
as /api/pty), via a read-only getApiRequestProfile() accessor.
/api/audio/transcribe, /api/audio/speak, /api/audio/elevenlabs/voices, and
the /api/audio/speak-stream WebSocket resolved TTS/STT config from the
dashboard's own HERMES_HOME regardless of the active profile, so a
non-default profile's voice settings were silently ignored. Give all four
the same optional profile param as the rest of the dashboard surface,
entering _config_profile_scope (await-safe, config-only — the audio paths
touch no skills globals) inside their worker threads.
Backend half of the desktop fix; completes the renderer-side profileScoped()
threading. Fixes#53441#45506#66012#64057.
Generated media often arrives as image markdown (). The chat
markdown renderer maps the img element to MarkdownImage, which renders a raw
<img>; for a video/audio source the browser cannot paint it and shows a
broken-image icon even though the file is valid and plays fine. Images worked
only because <img src=...png> is valid markup.
Route video/audio sources in MarkdownImage to the existing MediaAttachment,
which already picks the correct <video>/<audio> element (streaming protocol +
open-externally fallback) by media kind. The intended MEDIA:-tag -> #media:
link path is unaffected; this only fixes the bare image-markdown case.
Since #57944 the image path is built on hooks (async src resolution, failure
and loading states), so the routing check cannot be a plain early return
inside it -- it would have to sit after every hook call and would still fire
an image resolve for media never rendered as an image. MarkdownImage is
therefore a thin hookless router in front of MarkdownImageContent, which is
the previous body unchanged.
The regression tests join the existing markdown-text.media.test.tsx added by
#57944 rather than replacing it.
Fixes#40896
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vU45SEvcxLkVeyyEE9gJ2
transcribeAudio/speakText/getElevenLabsVoices called the backend without
profileScoped(), so for a non-default profile they hit the default backend
config instead of the active profile - playback used the wrong TTS/voice
even though settings were saved to the active profile config.
Fixes#53441
Two in-house micro-fixes from issue triage:
- #49883: voice.beep_enabled gates in cli.py and hermes_cli/voice.py used
bool() on the config value, so a quoted YAML string like "false" or
"off" kept beeps on. Route through utils.is_truthy_value.
- #18432: AudioRecorder.start() collapsed OSError from _import_audio into
the 'pip install sounddevice numpy' hint — but OSError means the
PortAudio SHARED LIBRARY is missing, which pip cannot fix. Mirror
detect_audio_environment's system-package hint (libportaudio2 /
brew portaudio / Termux pkg install portaudio) on that path.
Fixes#49883Fixes#18432
Typed messages sent while voice mode was active were also getting the
'[Voice input — respond concisely...]' API-local prefix, because the gate
checked only self._voice_mode. Route STT transcripts through a
_VoiceInputMessage sentinel in _pending_input (both the PTT/continuous
transcription path and the barge-in utterance path), unwrap it in
process_loop, and thread voice_input= through chat() so the prefix applies
only to genuinely voice-transcribed messages.
Re-cut of PR #65961 (@webtecnica) — the original diff had the sentinel
class embedded inside __init__'s docstring. Credit also to the earliest
route-by-origin attempt in PR #11744 (@KeroZelvin).
Fixes#65827Closes#65961Closes#11744
The quickstart voice-mode section used `uv pip install -e ".[voice]"`
which fails on a fresh curl-installed setup because no virtual environment
is active. Use `--python ./venv/bin/python` to target the installer-created
venv explicitly, matching the curl installer layout.
Fixes#44364
When Hermes runs inside its bundled venv (~/.hermes/profiles/<name>/
hermes-agent/venv/), the bare 'pip install sounddevice numpy' hint in
_voice_capture_install_hint() told users to install into whichever
Python their shell resolves first — on macOS that is often the system
Python under Rosetta, a completely separate site-packages tree with
incompatible wheel arches.
Detect venv via sys.prefix != sys.base_prefix and return the full path
to the venv's pip binary (Path(sys.prefix)/bin/pip) when available.
Falls back to the bare hint outside a venv (e.g. bare-metal installs).
Also adds 'from pathlib import Path' which was the only new import.
Addresses review on #62601. Applies a single rule — no sounddevice for
audio OUTPUT on macOS (PortAudio/CoreAudio init triggers a
kTCCServiceMediaLibrary prompt) — consistently at all three output sites:
- play_audio_file: WAV playback (already routed to afplay) now uses the
shared _sounddevice_output_allowed() helper.
- play_beep: synthesize the tone with numpy only, then on macOS play it
via a temp WAV through afplay instead of sounddevice.
- stream_tts_to_speaker (tts_tool): on macOS, skip the sounddevice
OutputStream so playback falls through to the existing tempfile/afplay path.
Audio INPUT (recording) is untouched — it legitimately needs mic permission.
Tests: TestMacOSAudioOutputPolicy (voice_mode) proves WAV + beep routing
does not import sounddevice on Darwin and still uses it off Darwin;
test_tts_macos_output proves streaming TTS skips the OutputStream on Darwin.
Existing test_play_wav_via_sounddevice pinned to non-Darwin for determinism.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On macOS, initializing PortAudio/CoreAudio via sounddevice triggers a
kTCCServiceMediaLibrary permission dialog even when no media-library
access is needed. afplay already handles WAV (and every other format)
natively, so route macOS playback straight to it and keep sounddevice
for other platforms.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two test classes lock in the new detection contract:
1. TestTermuxApiAppInstalledProbeLadder — drives _termux_api_app_installed
with a fake subprocess.run dispatcher and walks each rung of the
ladder:
- non-Termux env returns False (no probes run),
- pm-confirms returns True (back-compat),
- pm-clean-miss → cmd-confirms returns True,
- pm-FileNotFoundError → cmd-confirms returns True,
- pm-TimeoutExpired → cmd-confirms returns True,
- pm-nonzero-exit → cmd-confirms returns True,
- both probes inconclusive + binary on PATH returns True (the
core #31015 case),
- both probes inconclusive + no binary returns False,
- both probes clean-miss returns False (the genuine "CLI without
app" case keeps the existing warning),
- case-insensitive package match for ROMs that capitalise differently.
2. TestDetectAudioEnvironmentTermuxFallback — end-to-end through
detect_audio_environment, asserting the misleading "Termux:API
Android app is not installed" warning no longer fires when probes
are inconclusive but the binary is on PATH (the user-reported #31015
symptom), AND that the warning still fires when probes can
conclusively report the app is missing.
Refs: NousResearch/hermes-agent#31015
\`pm list packages com.termux.api\` is the canonical way to detect the
Termux:API Android app, but on some devices it gives a false negative
even when the app is installed and \`termux-microphone-record\` runs
fine — the symptom reported in #31015 (\`/voice on\` complaining
"Termux:API Android app is not installed").
Replace the single probe with a graded strategy:
1. Try \`pm list packages com.termux.api\` (current behaviour).
2. If \`pm\` isn't on PATH or returns non-zero, fall back to
\`cmd package list packages com.termux.api\` — the modern Android
API 28+ equivalent that's present on devices where \`pm\` is gone.
3. If both probes are inconclusive (binary missing, permission
denied, timeout, or non-zero exit) and \`termux-microphone-record\`
is on PATH, trust the binary. The CLI ships in the \`termux-api\`
package which is essentially only useful with the Android app
installed; users who installed it deliberately almost always have
the app too.
Polarity matters: a false negative on this gate blocks \`/voice on\`
entirely (the user-reported symptom), while a false positive only
surfaces a precise runtime error from the binary itself when it
tries to talk to the missing app — strictly more actionable.
The clean-probe-but-no-package case still returns False, so the
existing "Termux:API CLI installed without the app" warning still
fires when the package manager *can* tell us the app is missing.
Refs: NousResearch/hermes-agent#31015
Ports #63768 forward onto current main per teknium1's review.
On WSL2 without a PulseAudio bridge, ffplay and aplay have no audio
device and TTS playback silently fails (issue #17608). When
powershell.exe and ffmpeg are available, convert the audio to a
uniquely-named WAV in the Windows %TEMP% directory and play it via
Media.SoundPlayer.
Per review, this fixes two gaps in the original port:
1. Exit-status masking: the cleanup subshell was
'( ffmpeg && powershell ); rm -f wav' -- the shell's exit status is
the LAST command's (rm -f, which is always 0), so a real
ffmpeg/PowerShell failure could never be detected by the rc-checking
fallback logic added to the player loop. Now captures the real
status before cleanup and re-exits with it:
'( ffmpeg && powershell ); rc=0; rm -f wav; exit '.
2. The no-Pulse WSL gate in detect_audio_environment() still hard-blocked
voice mode entirely (input AND output) even when the PowerShell
fallback made TTS output viable. Added _wsl_powershell_tts_available()
and use it to downgrade the WSL-without-Pulse case from a hard
'warnings' block to a non-blocking 'notices' entry when the fallback
is available -- the same PulseAudio-bridge recording guidance is still
surfaced (mic capture genuinely still needs it), it just no longer
blocks /voice on for TTS-only usage. cli.py's existing
env_check['available'] gate needed no changes since it already
respects this flag.
Also fixed the flaky uniqueness test (the original asserted
len(filenames) >= 2, which passed trivially on zero captured
filenames) and added a real fallback-triggering regression test for
the exit-status fix.
10 new/fixed tests pass in TestWSL2PowerShellFallback and the new
TestWSLAudioEnvironmentGate; 80/80 in the full tests/tools/test_voice_mode.py file.
WSLg RDP audio has two issues causing crackling:
1. systemd-timesyncd clock adjustments jitter PulseAudio timing
(microsoft/wslg#1257) — user action: stop the service
2. Cold-start RDP connection drops first ~100ms of audio packets
before the virtual channel stabilises
Fix (automated, WSL-only):
- Detect WSL via /proc/version 'microsoft' marker
- Prepend 100ms silence + apply 100ms fade-in to audio
- Append 50ms silence tail for clean stream teardown
- Set blocksize=4096 (default auto ~1024 is too small for RDP)
- All in a single continuous sd.play() buffer
Non-WSL paths unchanged.
Closes#38893
AudioRecorder hard-codes SAMPLE_RATE (16 kHz) when opening the input
stream, but some capture devices (e.g. USB microphones exposed through
ALSA hw) reject 16 kHz outright — sd.InputStream fails with
PaErrorCode -9997 (Invalid sample rate) and voice recording is broken.
Query the default input device for its native default_samplerate at
recording start and open the stream / write the WAV at that rate,
falling back to the Whisper-friendly 16 kHz constant when the backend
does not expose a usable rate. STT providers accept standard WAV rates,
so downstream transcription is unaffected.
Closes#55908. The CLI voice-mode beep amplitude is hardcoded at 0.3 inside
tools.voice_mode:play_beep(), which makes the record start/stop cues too
quiet on low-volume systems and headphones. Users couldn't adjust it
without editing source.
Move the literal into a configurable voice.beep_volume setting (clamped to
0.0-1.0, default 0.3 to preserve prior behaviour). The new
_get_beep_volume() helper reads via the same load_config() pattern used by
cli.py's _voice_beeps_enabled() and hermes_cli/voice.py's _beeps_enabled(),
keeps bools / out-of-range / non-numeric / NaN values safely on the default,
and falls back silently if config can't load so the audio cue never breaks
the voice loop on a degenerate config.yaml.
Covered by tests/tools/test_voice_mode.py:
- TestGetBeepVolume (12 cases: missing key, custom value, boundary 0.0/1.0,
out-of-range clamp, type coercion, bool guard, NaN guard, exception
guard, dict-typed voice section)
- TestPlayBeepVolumeWiring (guards against re-introducing a hardcoded 0.3
literal in play_beep)
Docs: website/docs/user-guide/configuration.md mentions the new key.
Other locale translations (zh-Hans etc.) intentionally untouched —
handled by the regular i18n sync pipeline as a separate change.
No change in default behaviour: existing users hear exactly the same beep.
