hermes-agent/tools/wake_word.py
Teknium 0f64557c06 fix(wake): coerce dead onnx->tflite on macOS ARM64; clear stale voice turn-timeout
Two follow-ups from the voice PR (#70509).

1. macOS ARM64 onnx migration. Existing users who pinned
   openwakeword.inference_framework=onnx before the tflite fix landed kept a
   wake word that arms but never fires (ONNX's embedding model is broken on
   Apple Silicon, upstream #336). New resolve_inference_framework() honors an
   explicit framework everywhere ONNX actually works, but coerces the one
   provably-dead combination (explicit onnx + macOS ARM64) to tflite with a
   one-time warning. No config mutation; empty still falls back to the platform
   default. Both read sites (engine init + requirements check) route through
   the shared resolver.

2. Voice turn-timeout leak. Each listen cycle reassigned turnTimeoutRef
   without clearing the prior 60s timer, so a stale timer from an earlier
   cycle could fire handleTurn() mid-way through a later listen — after enough
   idle re-listens this wedged the loop into a non-re-arming state (the
   'voice chat deactivates after ~a minute' report). Clear before re-arm.

Tests: 64 wake tests (added onnx-coercion / intel-kept / tflite-kept /
empty-default cases; updated the stale 'explicit onnx kept on ARM64' test
that encoded the old broken behavior), 39 desktop voice/wake vitest, tsc +
eslint clean.
2026-07-28 19:40:25 -07:00

1157 lines
44 KiB
Python

"""Wake-word ("Hey Hermes") detection — hands-free session trigger.
A lightweight, always-on hotword listener that fires a callback when a wake
phrase is spoken — the "Hey Siri" / "Alexa" pattern. Shared by the CLI, TUI, and
desktop GUI (one of them owns it, gated by ``wake_surface_enabled``): say the
wake word, Hermes opens a fresh session and captures voice via the existing
pipeline, then answers.
Three engines, all fully on-device (no audio leaves the machine for detection):
* **openwakeword** (default, free, no API key) — loads an ONNX model. Defaults
to the bundled "hey hermes" model (``tools/wakewords/``) so the wake word
works out of the box; or point ``wake_word.openwakeword.model`` at a built-in
name (``hey_jarvis``, ``alexa``, …) or a custom ``.onnx`` for another phrase.
* **sherpa** (free, no API key, open vocabulary) — sherpa-onnx keyword
spotting. Detects ANY typed phrase with no training: set
``wake_word.phrase`` and the phrase is tokenized at runtime against a small
streaming zipformer model (~13 MB English model, one-time download).
* **porcupine** (premium) — Picovoice's engine. Needs ``PORCUPINE_ACCESS_KEY``;
supports built-in keywords and custom ``.ppn`` files from the Picovoice
Console.
Audio capture reuses the same 16 kHz mono int16 ``sounddevice`` path as voice
mode. The detector runs on its own daemon thread; callers ``pause()`` it while a
voice turn holds the microphone and ``resume()`` it once the system is idle
again (two input streams on one device is unreliable cross-platform).
Nothing here mutates agent context or the prompt cache — on wake we hand a plain
string to the caller, exactly like a voice transcript.
"""
from __future__ import annotations
import logging
import os
import sys
import threading
import time
from pathlib import Path
from typing import Any, Callable, Dict, Optional
logger = logging.getLogger(__name__)
# 16 kHz mono int16 — Whisper-native and what both engines expect.
SAMPLE_RATE = 16000
# Minimum gap between two consecutive wake fires, so one "hey hermes" can't
# retrigger across several frames while the caller is still reacting.
_FIRE_COOLDOWN_SECONDS = 2.0
_START_TIMEOUT_SECONDS = 5.0
# Ambient-speech rejection: openWakeWord scores one ~80ms frame at a time, and a
# stray phoneme in background conversation can spike a single frame over the
# threshold. A real utterance of the phrase holds the score high across several
# consecutive frames, so we require N-in-a-row above threshold before firing.
# This is the primary lever against unintended triggers on ambient talk.
_DEFAULT_CONFIRMATION_FRAMES = 3
# Dead-mic detection: an int16 stream whose peak stays at/below this for this
# many consecutive seconds is flagged as silent. macOS grants the *app* mic
# permission per-process — a backend spawned without the entitlement gets a
# "working" CoreAudio stream that delivers zeros forever, so the listener
# looks armed but can never hear the phrase.
_SILENCE_PEAK = 10
_SILENCE_ALERT_SECONDS = 10
class WakeWordInUse(RuntimeError):
"""Raised when another surface or process owns the wake-word listener."""
# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------
_DEFAULTS: Dict[str, Any] = {
"enabled": False,
"surface": "auto",
"provider": "openwakeword",
"phrase": "hey hermes",
"sensitivity": 0.6,
"confirmation_frames": _DEFAULT_CONFIRMATION_FRAMES,
"start_new_session": True,
}
# Bundled "hey hermes" model (tools/wakewords/) — the default, so the wake word
# works out of the box. Config names in _ALIASES resolve to it, not a built-in.
_BUNDLED_MODEL_NAME = "hey_hermes"
_BUNDLED_MODEL_ALIASES = frozenset({"", "hey_hermes", "hey hermes", "hermes"})
def _bundled_wakeword_path(framework: str = "onnx") -> str:
"""Path to the shipped hey_hermes model (.onnx/.tflite) for ``framework``."""
ext = "tflite" if str(framework).strip().lower() == "tflite" else "onnx"
return os.path.join(os.path.dirname(__file__), "wakewords", f"{_BUNDLED_MODEL_NAME}.{ext}")
def _is_macos_arm64() -> bool:
import platform
return sys.platform == "darwin" and platform.machine() == "arm64"
def default_inference_framework() -> str:
"""The openWakeWord backend to use on this platform.
openWakeWord's ONNX backend produces near-zero scores on macOS ARM64 — its
shared *embedding* model is the broken stage (the melspectrogram front-end
and the wake classifier both match tflite exactly). The detector arms, the
microphone works, and no phrase can ever cross the threshold. Prefer the
tflite backend there; ONNX stays the default everywhere else.
Upstream: https://github.com/dscripka/openWakeWord/issues/336
"""
return "tflite" if _is_macos_arm64() else "onnx"
_warned_onnx_coerced = False
def resolve_inference_framework(cfg: Dict[str, Any]) -> str:
"""Resolve the effective openWakeWord backend from config.
Honors an explicit ``openwakeword.inference_framework`` — EXCEPT the one
combination that is provably dead: an explicit ``onnx`` on macOS ARM64,
where ONNX's embedding model never lets a phrase cross threshold (upstream
#336). Existing macOS users who pinned ``onnx`` before the tflite fix landed
would otherwise keep a wake word that arms but never fires. Coerce that one
case to tflite (with a one-time warning) instead of silently shipping a dead
ear. Every other explicit value is respected as-is; empty falls back to the
platform default.
"""
global _warned_onnx_coerced
sub = cfg.get("openwakeword") if isinstance(cfg.get("openwakeword"), dict) else {}
framework = str(sub.get("inference_framework") or "").strip().lower()
if not framework:
return default_inference_framework()
if framework == "onnx" and _is_macos_arm64():
if not _warned_onnx_coerced:
_warned_onnx_coerced = True
logger.warning(
"wake: openwakeword.inference_framework='onnx' is set but ONNX's "
"embedding model never fires on macOS ARM64 (openWakeWord #336) — "
"using tflite instead. Set inference_framework to '' (auto) or "
"'tflite' in config.yaml to silence this."
)
return "tflite"
return framework
def ensure_tflite_runtime() -> bool:
"""Make ``import tflite_runtime.interpreter`` resolve, returning success.
openWakeWord hardcodes that import but only declares ``tflite-runtime`` for
``platform_system == "Linux"``; on macOS the equivalent wheel is
``ai-edge-litert``. Alias the module so the upstream import succeeds. The
alias is process-local — nothing is written to site-packages.
"""
try:
import tflite_runtime.interpreter # noqa: F401
return True
except ImportError:
pass
try:
from ai_edge_litert import interpreter as _litert # type: ignore[import-not-found]
except ImportError:
return False
import types
pkg = types.ModuleType("tflite_runtime")
pkg.__path__ = [] # type: ignore[attr-defined] # mark as package
sys.modules.setdefault("tflite_runtime", pkg)
sys.modules["tflite_runtime.interpreter"] = _litert
logger.debug("wake word: bridged tflite_runtime -> ai_edge_litert")
return True
def load_wake_word_config() -> Dict[str, Any]:
"""Return the ``wake_word`` config section, shape-guarded to a dict."""
try:
from hermes_cli.config import load_config
cfg = load_config().get("wake_word")
except Exception:
cfg = None
return cfg if isinstance(cfg, dict) else {}
def _get(cfg: Dict[str, Any], key: str) -> Any:
val = cfg.get(key, _DEFAULTS.get(key))
return _DEFAULTS.get(key) if val is None else val
def _provider(cfg: Dict[str, Any]) -> str:
return str(_get(cfg, "provider")).strip().lower() or "openwakeword"
def _sensitivity(cfg: Dict[str, Any]) -> float:
raw = _get(cfg, "sensitivity")
try:
s = float(raw)
except (TypeError, ValueError):
s = float(_DEFAULTS["sensitivity"])
return min(max(s, 0.0), 1.0)
def _confirmation_frames(cfg: Dict[str, Any]) -> int:
"""How many consecutive over-threshold frames are required to fire.
``1`` restores the old single-frame behaviour; higher values reject
ambient-speech blips at the cost of a few tens of ms of extra latency.
Clamped to a sane 1..10.
"""
raw = _get(cfg, "confirmation_frames")
try:
n = int(raw)
except (TypeError, ValueError):
n = _DEFAULT_CONFIRMATION_FRAMES
return min(max(n, 1), 10)
def wake_phrase(cfg: Optional[Dict[str, Any]] = None) -> str:
"""Human-facing wake phrase label (purely cosmetic; engine keys detection)."""
cfg = cfg if cfg is not None else load_wake_word_config()
return str(_get(cfg, "phrase")) or "hey hermes"
def wake_surface_enabled(surface: str, cfg: Optional[Dict[str, Any]] = None) -> bool:
"""Should ``surface`` (``cli`` / ``tui`` / ``gui``) host the listener?
True when the wake word is enabled and the configured ``surface`` is either
``auto`` or this exact surface. ``auto`` makes a surface eligible; the
process/machine ownership lock still permits only the first claimant.
"""
cfg = cfg if cfg is not None else load_wake_word_config()
if not cfg.get("enabled"):
return False
want = str(_get(cfg, "surface")).strip().lower() or "auto"
return want == "auto" or want == surface.strip().lower()
# ---------------------------------------------------------------------------
# Multi-profile phrase enrollment (open-vocabulary routing)
# ---------------------------------------------------------------------------
def _active_profile_name() -> str:
try:
from hermes_cli.profiles import get_active_profile_name
return get_active_profile_name() or "default"
except Exception:
return "default"
def enrolled_profile_phrases() -> Dict[str, str]:
"""Map ``profile name -> wake phrase`` for every wake-enabled profile.
Reads each profile's own ``config.yaml`` raw (cheap, no full config merge).
A profile is enrolled when its ``wake_word.enabled`` is truthy; its phrase
defaults to ``"hey <profile>"`` when unset. Used by the sherpa engine to
listen for every enrolled profile's phrase at once and route the wake to
the matching profile. Best-effort: unreadable profiles are skipped.
"""
phrases: Dict[str, str] = {}
try:
import yaml
from hermes_cli.profiles import get_profile_dir, list_profiles
for info in list_profiles():
name = getattr(info, "name", None) or str(info)
try:
cfg_path = Path(get_profile_dir(name)) / "config.yaml"
raw = yaml.safe_load(cfg_path.read_text(encoding="utf-8")) or {}
wc = raw.get("wake_word") or {}
if not isinstance(wc, dict) or not wc.get("enabled"):
continue
phrase = str(wc.get("phrase") or f"hey {name}").strip()
if phrase:
phrases[name] = phrase
except Exception:
continue
except Exception:
pass
return phrases
# ---------------------------------------------------------------------------
# Audio capture (lazy — never import sounddevice at module load)
# ---------------------------------------------------------------------------
def _import_audio():
import numpy as np
import sounddevice as sd
return sd, np
def _audio_available() -> bool:
try:
_import_audio()
return True
except (ImportError, OSError):
return False
# ---------------------------------------------------------------------------
# Engines
# ---------------------------------------------------------------------------
class _Engine:
"""Minimal hotword-engine contract: feed int16 frames, get a bool."""
frame_length: int = 1280 # 80 ms at 16 kHz
#: Optional (matched phrase, profile name) of the most recent fire.
#: Multi-phrase engines (sherpa) set this for profile routing; the
#: single-phrase engines leave it None (callers fall back to the
#: configured phrase / active profile).
last_match: Optional[tuple[str, str]] = None
def process(self, frame) -> bool: # frame: 1-D int16 ndarray
raise NotImplementedError
def reset(self) -> None:
"""Clear any internal audio/feature buffer (called on every (re)start)."""
pass
def close(self) -> None:
pass
def _looks_like_path(value: str) -> bool:
return (
os.sep in value
or value.endswith((".onnx", ".tflite", ".ppn"))
or os.path.exists(value)
)
class _OpenWakeWordEngine(_Engine):
"""openWakeWord — free, local ONNX hotword detection."""
# openWakeWord recommends 80 ms frames (1280 samples) for efficiency.
frame_length = 1280
def __init__(self, cfg: Dict[str, Any]):
from tools import lazy_deps
lazy_deps.ensure("wake.openwakeword", prompt=False)
import openwakeword
from openwakeword.model import Model
sub = cfg.get("openwakeword") if isinstance(cfg.get("openwakeword"), dict) else {}
model_ref = str(sub.get("model") or _BUNDLED_MODEL_NAME).strip()
framework = resolve_inference_framework(cfg)
# openWakeWord returns a 0..1 score per frame; sensitivity IS the raw
# threshold a score must clear. Higher = stricter (fewer false fires).
# Default 0.6 sits above openWakeWord's permissive 0.5 baseline, which
# let near-misses like "hey hor" through.
self._threshold = _sensitivity(cfg)
self._confirm_needed = _confirmation_frames(cfg)
self._confirm_streak = 0
# openWakeWord silently downgrades tflite -> onnx when no tflite runtime
# imports (model.py). On macOS ARM64 that lands on the backend whose
# embedding model is broken, so the listener would arm and never fire.
# Install + bridge the runtime first, and refuse the downgrade rather
# than ship a dead ear.
if framework == "tflite" and not ensure_tflite_runtime():
# Same lazy-install contract as every other backend; the platform
# gate lives here because dep specs can't carry PEP 508 markers.
try:
lazy_deps.ensure("wake.openwakeword.tflite", prompt=False)
except Exception as e:
logger.debug("wake word: tflite runtime install failed: %s", e)
if not ensure_tflite_runtime():
if _is_macos_arm64():
raise RuntimeError(
"The wake word needs the tflite backend on this Mac, but its "
"runtime is missing. Install it with: pip install ai-edge-litert"
)
logger.warning("wake word: no tflite runtime available — falling back to onnx")
framework = "onnx"
# Default (or explicit "hey_hermes") → the bundled model; a built-in name
# or custom path is used as-is.
if model_ref.lower() in _BUNDLED_MODEL_ALIASES:
model_ref = _bundled_wakeword_path(framework)
# openWakeWord needs its shared feature models (melspectrogram + embedding)
# for ANY model — download_models() fetches those first on every call, so a
# custom path must call it too, else a fresh install crashes on a missing
# melspectrogram.onnx. A built-in name additionally pulls that pretrained
# model; a path matches nothing in the catalog and is a no-op beyond base.
try:
openwakeword.utils.download_models([model_ref])
except Exception as e: # pragma: no cover - network/path dependent
logger.debug("openwakeword model download skipped: %s", e)
models = [model_ref]
self._model = Model(wakeword_models=models, inference_framework=framework)
self._labels = list(self._model.models.keys())
def process(self, frame) -> bool:
scores = self._model.predict(frame)
over = any(score >= self._threshold for score in scores.values())
# Require N consecutive over-threshold frames: a real phrase holds the
# score high across frames, a stray ambient phoneme spikes just one.
if over:
self._confirm_streak += 1
if self._confirm_streak >= self._confirm_needed:
self._confirm_streak = 0
return True
return False
self._confirm_streak = 0
return False
def reset(self) -> None:
# Clears openWakeWord's rolling feature/prediction buffer so stale audio
# captured before a pause can't re-fire the moment we resume.
self._confirm_streak = 0
try:
self._model.reset()
except Exception:
pass
def close(self) -> None:
self.reset()
# sherpa-onnx open-vocabulary KWS model: a small streaming zipformer
# transducer. English (GigaSpeech); one-time download, cached under
# HERMES_HOME. Keywords are typed phrases tokenized at RUNTIME — no
# training step, unlike openWakeWord/Porcupine custom models.
_SHERPA_KWS_MODEL_URL = (
"https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/"
"sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.tar.bz2"
)
_SHERPA_KWS_MODEL_DIR = "sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01"
def _sherpa_model_root() -> Path:
from hermes_constants import get_hermes_home
return get_hermes_home() / "cache" / "wakewords"
def _ensure_sherpa_model(root: Optional[Path] = None) -> Path:
"""Download + unpack the sherpa KWS model once; return its directory."""
root = root or _sherpa_model_root()
target = root / _SHERPA_KWS_MODEL_DIR
if (target / "tokens.txt").exists():
return target
import tarfile
import urllib.request
root.mkdir(parents=True, exist_ok=True)
archive = root / f"{_SHERPA_KWS_MODEL_DIR}.tar.bz2"
logger.info("wake word: downloading sherpa KWS model (one-time, ~13 MB)")
urllib.request.urlretrieve(_SHERPA_KWS_MODEL_URL, archive) # noqa: S310
with tarfile.open(archive, "r:bz2") as tf:
tf.extractall(root, filter="data")
archive.unlink(missing_ok=True)
if not (target / "tokens.txt").exists():
raise RuntimeError(f"sherpa KWS model unpack failed: {target}")
return target
class _SherpaKwsEngine(_Engine):
"""sherpa-onnx open-vocabulary keyword spotting — any typed phrase, zero training.
The configured ``wake_word.phrase`` is BPE-tokenized at runtime against the
model's vocabulary, so "hey hermes", "hey coder", or any other phrase works
immediately. Here ``phrase`` is DETECTION config, not a cosmetic label.
"""
# sherpa's streaming zipformer consumes arbitrary chunk sizes; 1280
# samples (80 ms) matches the shared capture path.
frame_length = 1280
def __init__(self, cfg: Dict[str, Any]):
from tools import lazy_deps
lazy_deps.ensure("wake.sherpa", prompt=False)
import sherpa_onnx
from sherpa_onnx import text2token
sub = cfg.get("sherpa") if isinstance(cfg.get("sherpa"), dict) else {}
model_dir = str(sub.get("model_dir") or "").strip()
d = Path(model_dir) if model_dir else _ensure_sherpa_model()
if not (d / "tokens.txt").exists():
raise RuntimeError(f"sherpa KWS model not found at {d}")
# Phrase set: this profile's own phrase, plus — when profile routing is
# on — every other wake-enabled profile's phrase, so ONE listener can
# wake any profile ("hey hermes" / "hey coder" / ...). display-name →
# profile is kept for routing the match back.
phrase = str(_get(cfg, "phrase") or "hey hermes").strip()
own_profile = _active_profile_name()
phrase_map: Dict[str, str] = {phrase: own_profile}
if bool(cfg.get("profile_routing", True)):
for prof, p in enrolled_profile_phrases().items():
phrase_map.setdefault(p.strip(), prof)
phrases = list(phrase_map)
# Runtime tokenization of the arbitrary phrases — the open-vocab core.
tokens = text2token(
[p.upper() for p in phrases],
tokens=str(d / "tokens.txt"),
tokens_type="bpe",
bpe_model=str(d / "bpe.model"),
)
import tempfile
# sherpa keyword entries reject spaces in the @display-name; underscore
# them and map display → profile for match routing.
self._display_to_profile: Dict[str, str] = {}
kw = tempfile.NamedTemporaryFile(
mode="w", suffix=".txt", prefix="hermes-kws-", delete=False, encoding="utf-8"
)
for p, toks in zip(phrases, tokens):
display = p.upper().replace(" ", "_")
self._display_to_profile[display] = phrase_map[p]
kw.write(" ".join(toks) + f" @{display}\n")
kw.close()
self._keywords_file = kw.name
#: (phrase display name, profile) of the most recent fire, for routing.
self.last_match: Optional[tuple[str, str]] = None
# Map the shared 0..1 sensitivity onto sherpa's keywords_threshold.
# 0.5 lands exactly on sherpa's recommended default (0.25); live TTS
# matrix testing showed our previous stricter mapping (0.35) missed
# ~12% of true positives while 0.25 held zero false fires.
threshold = 0.05 + 0.4 * _sensitivity(cfg)
def _model_file(pattern: str) -> str:
hits = sorted(d.glob(pattern))
if not hits:
raise RuntimeError(f"sherpa KWS model file missing: {d}/{pattern}")
return str(hits[0])
self._spotter = sherpa_onnx.KeywordSpotter(
tokens=str(d / "tokens.txt"),
encoder=_model_file("encoder-*[!8].onnx"),
decoder=_model_file("decoder-*[!8].onnx"),
joiner=_model_file("joiner-*[!8].onnx"),
keywords_file=self._keywords_file,
keywords_threshold=threshold,
num_threads=1,
)
self._stream = self._spotter.create_stream()
def process(self, frame) -> bool:
import numpy as np
samples = np.asarray(frame, dtype=np.float32) / 32768.0
self._stream.accept_waveform(SAMPLE_RATE, samples)
fired = False
while self._spotter.is_ready(self._stream):
self._spotter.decode_stream(self._stream)
result = self._spotter.get_result(self._stream)
if result:
fired = True
display = str(result)
self.last_match = (
display.replace("_", " ").lower(),
self._display_to_profile.get(display, ""),
)
# Reset decoder state so one utterance can't fire repeatedly.
self._spotter.reset_stream(self._stream)
return fired
def reset(self) -> None:
# Fresh stream drops all buffered audio/decoder state (pause → resume
# must not re-fire on stale audio).
try:
self._stream = self._spotter.create_stream()
except Exception:
pass
def close(self) -> None:
try:
os.unlink(self._keywords_file)
except OSError:
pass
class _PorcupineEngine(_Engine):
"""Picovoice Porcupine — premium, on-device, needs an access key."""
def __init__(self, cfg: Dict[str, Any]):
from tools import lazy_deps
lazy_deps.ensure("wake.porcupine", prompt=False)
import pvporcupine
access_key = (os.getenv("PORCUPINE_ACCESS_KEY") or "").strip()
if not access_key:
raise RuntimeError(
"Porcupine wake word requires PORCUPINE_ACCESS_KEY "
"(get a free key at https://console.picovoice.ai)."
)
sub = cfg.get("porcupine") if isinstance(cfg.get("porcupine"), dict) else {}
keyword = str(sub.get("keyword") or "jarvis").strip()
# Porcupine's `sensitivities` runs the OPPOSITE way to our shared knob:
# per Picovoice, higher = more true positives AND more false alarms
# (looser). Our config contract is "higher = stricter" everywhere, so
# invert it here to keep one consistent meaning across all engines.
porcupine_sensitivity = 1.0 - _sensitivity(cfg)
kwargs: Dict[str, Any] = {"access_key": access_key, "sensitivities": [porcupine_sensitivity]}
if _looks_like_path(keyword):
kwargs["keyword_paths"] = [keyword]
else:
kwargs["keywords"] = [keyword]
self._porcupine = pvporcupine.create(**kwargs)
self.frame_length = self._porcupine.frame_length
def process(self, frame) -> bool:
# pvporcupine wants a plain list/sequence of int16 samples.
return self._porcupine.process(frame) >= 0
def close(self) -> None:
try:
self._porcupine.delete()
except Exception:
pass
def _build_engine(cfg: Dict[str, Any]) -> _Engine:
provider = _provider(cfg)
if provider == "porcupine":
return _PorcupineEngine(cfg)
if provider in ("sherpa", "sherpa-onnx", "kws", "open"):
return _SherpaKwsEngine(cfg)
if provider in ("openwakeword", "oww", "local"):
return _OpenWakeWordEngine(cfg)
raise ValueError(f"Unknown wake_word provider: {provider!r}")
# ---------------------------------------------------------------------------
# Requirements probe (for /wake status + enable path)
# ---------------------------------------------------------------------------
def _stt_ready() -> bool:
"""Is a speech-to-text provider configured and enabled?
A wake without STT arms the mic but every captured utterance dies at
transcription — a useless (and confusing) experience. Same standard as
voice mode's ``check_voice_requirements``: enabled + a real provider.
"""
try:
from tools.transcription_tools import _get_provider, _load_stt_config, is_stt_enabled
stt_config = _load_stt_config()
return is_stt_enabled(stt_config) and _get_provider(stt_config) != "none"
except Exception:
return False
def _tts_ready() -> bool:
"""Can the configured text-to-speech provider run (or install at first use)?
The wake flow is fully hands-free (wake → speak → hear the reply); without
TTS the reply is silent and the loop is pointless.
PROBE, not an installer: ``check_tts_requirements`` lazily pip-installs the
provider SDK via ``_import_*`` → ``lazy_deps.ensure`` — running that inside
a status poll froze wake.status for the length of a pip install (and a
failed install marked the wake word unavailable, unmounting the desktop
ear). When the provider's deps aren't installed yet, "installable at first
use" counts as ready and we never touch pip from here.
"""
try:
from tools.tts_tool import _get_provider, _load_tts_config
provider = _get_provider(_load_tts_config())
except Exception:
return False
_LAZY_TTS_FEATURES = {
"edge": "tts.edge",
"elevenlabs": "tts.elevenlabs",
"mistral": "tts.mistral",
}
feature = _LAZY_TTS_FEATURES.get(provider)
if feature is not None:
try:
from tools import lazy_deps
if not lazy_deps.is_available(feature):
# Not installed: ready iff it can install at first speak.
return lazy_deps._allow_lazy_installs()
except Exception:
return False
try:
from tools.tts_tool import check_tts_requirements
return bool(check_tts_requirements())
except Exception:
return False
def check_wake_word_requirements(cfg: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
"""Report whether wake-word detection can run, with a remediation hint."""
cfg = cfg if cfg is not None else load_wake_word_config()
provider = _provider(cfg)
from tools import lazy_deps
if provider == "porcupine":
feature = "wake.porcupine"
elif provider in ("sherpa", "sherpa-onnx", "kws", "open"):
feature = "wake.sherpa"
else:
feature = "wake.openwakeword"
deps_ok = lazy_deps.is_available(feature)
lazy_ok = lazy_deps._allow_lazy_installs()
# The audio probe imports sounddevice + numpy — two of the very packages
# the lazy installer would fetch — so it can only be trusted once the
# feature's deps are installed. On a fresh install (deps missing, lazy
# installs allowed) we defer the mic check: the engine constructors call
# ``lazy_deps.ensure()`` and the stream-open surfaces any real audio
# problem. Gating ``available`` on the probe here made the lazy-install
# path unreachable (the probe always failed before ensure() could run).
audio_ok = _audio_available() if deps_ok else False
key_ok = True
# The full wake loop is wake → record → STT → agent → TTS. Arming without
# either end configured gives a mic that hears you and then does nothing
# the user can perceive — refuse with a pointer instead.
stt_ok = _stt_ready()
tts_ok = _tts_ready()
hint = ""
# The tflite backend needs a runtime openWakeWord doesn't declare off Linux.
# Report it as a real remediation instead of arming a detector that can't fire.
tflite_ok = True
if provider not in ("porcupine", "sherpa", "sherpa-onnx", "kws", "open"):
framework = resolve_inference_framework(cfg)
if framework == "tflite":
tflite_ok = ensure_tflite_runtime() or lazy_deps.is_available("wake.openwakeword.tflite") or lazy_ok
if provider == "porcupine" and not (os.getenv("PORCUPINE_ACCESS_KEY") or "").strip():
key_ok = False
hint = "Set PORCUPINE_ACCESS_KEY (free key at https://console.picovoice.ai)."
elif not deps_ok and not lazy_ok:
hint = lazy_deps.feature_install_command(feature) or ""
elif not tflite_ok:
hint = "The wake word needs the tflite runtime on this Mac: pip install ai-edge-litert"
elif deps_ok and not audio_ok:
hint = "Microphone capture needs sounddevice + numpy and a working audio device."
elif not stt_ok or not tts_ok:
missing = " and ".join(
name for name, ok in (("speech-to-text", stt_ok), ("text-to-speech", tts_ok)) if not ok
)
hint = (f"Wake word needs {missing} configured — run `hermes tools` "
f"(Voice section) or see the voice-mode docs.")
return {
"available": key_ok and stt_ok and tts_ok and tflite_ok
and ((deps_ok and audio_ok) or (not deps_ok and lazy_ok)),
"provider": provider,
"deps_available": deps_ok,
"audio_available": audio_ok,
"access_key_set": key_ok,
"stt_available": stt_ok,
"tts_available": tts_ok,
"phrase": wake_phrase(cfg),
"hint": hint,
}
# ---------------------------------------------------------------------------
# Detector
# ---------------------------------------------------------------------------
class WakeWordDetector:
"""Background hotword listener. Fires ``on_wake()`` when the phrase is heard.
The engine is built once and kept alive across pause/resume; only the audio
stream + reader thread cycle, so toggling the mic for a voice turn is cheap.
"""
def __init__(self, engine: _Engine, on_wake: Callable[[], None],
cooldown: float = _FIRE_COOLDOWN_SECONDS,
on_failure: Optional[Callable[["WakeWordDetector"], None]] = None):
self.engine = engine
self.on_wake = on_wake
self.cooldown = cooldown
self.on_failure = on_failure
self._thread: Optional[threading.Thread] = None
self._stop = threading.Event()
self._callback_inflight = threading.Event()
self._last_fire = 0.0
self._lock = threading.Lock()
# True when the stream is open but every frame is (near-)silence — the
# classic macOS symptom of a backend process without mic permission:
# CoreAudio "succeeds" and delivers zeros forever. Surfaced via
# wake.status / /wake status so users can tell "armed" from "deaf".
self.audio_silent = False
self._silent_frames = 0
@property
def running(self) -> bool:
t = self._thread
return t is not None and t.is_alive()
def start(self) -> None:
"""Open the mic and begin listening. Idempotent."""
with self._lock:
if self._thread is not None and self._thread.is_alive():
return
self._stop.clear()
ready = threading.Event()
startup_errors: list[BaseException] = []
self._thread = threading.Thread(
target=self._run,
args=(ready, startup_errors),
daemon=True,
name="wake-word",
)
self._thread.start()
if not ready.wait(_START_TIMEOUT_SECONDS):
self._halt_thread()
raise TimeoutError("Timed out while opening the wake-word microphone.")
if startup_errors:
self._halt_thread()
raise RuntimeError("Failed to open the wake-word microphone.") from startup_errors[0]
# pause/resume keep the engine; stop tears it down.
def pause(self) -> None:
self._halt_thread()
def resume(self) -> None:
self.start()
def stop(self) -> None:
self._halt_thread()
self.engine.close()
def _halt_thread(self) -> None:
with self._lock:
self._stop.set()
t = self._thread
if t is not None and t is not threading.current_thread():
t.join(timeout=2.0)
if self._thread is t:
self._thread = None
def _dispatch_wake(self) -> None:
try:
self.on_wake()
except Exception as e:
logger.warning("wake word callback failed: %s", e)
finally:
self._callback_inflight.clear()
def _run(self, ready: threading.Event,
startup_errors: list[BaseException]) -> None:
try:
sd, _ = _import_audio()
except (ImportError, OSError) as e:
logger.error("wake word: audio libraries unavailable: %s", e)
startup_errors.append(e)
ready.set()
return
frame_length = self.engine.frame_length
try:
stream = sd.InputStream(
samplerate=SAMPLE_RATE,
channels=1,
dtype="int16",
blocksize=frame_length,
)
stream.start()
except Exception as e:
logger.error("wake word: failed to open microphone: %s", e)
startup_errors.append(e)
ready.set()
return
# Drop any buffered audio/feature state so a resume right after a voice
# turn can't immediately re-fire on audio captured before the pause (the
# wake → voice → resume → wake runaway loop).
try:
self.engine.reset()
except Exception:
pass
logger.info("wake word: listening (frame=%d, rate=%d)", frame_length, SAMPLE_RATE)
ready.set()
failed = False
# ~seconds of consecutive near-zero frames before we flag the stream
# as silent (macOS no-permission streams deliver zeros forever).
silent_alert_frames = max(1, int(_SILENCE_ALERT_SECONDS * SAMPLE_RATE / max(1, frame_length)))
try:
while not self._stop.is_set():
try:
data, _overflow = stream.read(frame_length)
except Exception as e:
logger.warning("wake word: stream read error: %s", e)
failed = not self._stop.is_set()
break
frame = data[:, 0] if getattr(data, "ndim", 1) == 2 else data
try:
peak = int(abs(frame).max()) if len(frame) else 0
except Exception:
peak = _SILENCE_PEAK + 1
if peak <= _SILENCE_PEAK:
self._silent_frames += 1
if self._silent_frames == silent_alert_frames:
self.audio_silent = True
logger.warning(
"wake word: mic delivers only silence (peak<=%d for %ds) — "
"on macOS check System Settings > Privacy & Security > "
"Microphone for the Hermes backend process",
_SILENCE_PEAK, _SILENCE_ALERT_SECONDS,
)
elif self._silent_frames:
if self.audio_silent:
logger.info("wake word: mic audio detected — stream healthy")
self._silent_frames = 0
self.audio_silent = False
try:
fired = self.engine.process(frame)
except Exception as e:
logger.debug("wake word: engine error: %s", e)
continue
if fired:
now = time.monotonic()
if now - self._last_fire >= self.cooldown:
self._last_fire = now
logger.info("wake word: phrase detected — firing callback")
if not self._callback_inflight.is_set():
self._callback_inflight.set()
threading.Thread(
target=self._dispatch_wake,
daemon=True,
name="wake-word-callback",
).start()
else:
logger.debug("wake word: detection within cooldown — ignored")
finally:
try:
stream.stop()
stream.close()
except Exception:
pass
logger.info("wake word: stream closed")
if failed and self.on_failure is not None:
self.on_failure(self)
# ---------------------------------------------------------------------------
# Process-wide singleton (mirrors hermes_cli.voice's continuous API)
# ---------------------------------------------------------------------------
_detector: Optional[WakeWordDetector] = None
_detector_owner: object | None = None
_detector_file_lock = None
_detector_lock = threading.Lock()
def _lock_path() -> Path:
from hermes_constants import get_default_hermes_root
return get_default_hermes_root() / "runtime" / "wake-word.lock"
def _acquire_machine_lock(path: Optional[Path] = None):
"""Acquire the cross-process microphone lease, or raise WakeWordInUse."""
lock_path = path or _lock_path()
lock_path.parent.mkdir(parents=True, exist_ok=True)
handle = open(lock_path, "a+b")
try:
if os.name == "nt":
import msvcrt
handle.seek(0, os.SEEK_END)
if handle.tell() == 0:
handle.write(b"\0")
handle.flush()
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1)
else:
import fcntl
fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
except (OSError, BlockingIOError) as e:
handle.close()
raise WakeWordInUse("Wake-word microphone is already owned.") from e
return handle
def _release_machine_lock(handle) -> None:
if handle is None:
return
try:
if os.name == "nt":
import msvcrt
handle.seek(0)
msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1)
else:
import fcntl
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)
except OSError:
pass
finally:
handle.close()
def _detector_failed(detector: WakeWordDetector) -> None:
"""Release ownership if the active microphone stream dies unexpectedly."""
global _detector, _detector_owner, _detector_file_lock
with _detector_lock:
if _detector is not detector:
return
lock_handle = _detector_file_lock
_detector = None
_detector_owner = None
_detector_file_lock = None
try:
detector.engine.close()
finally:
_release_machine_lock(lock_handle)
def start_listening(
on_wake: Callable[[], None],
*,
owner: object,
config: Optional[Dict[str, Any]] = None,
) -> WakeWordDetector:
"""Claim, build, and start the detector. Idempotent for the same owner.
Raises if engine construction fails (missing deps / access key / model);
callers should probe :func:`check_wake_word_requirements` first. A different
owner, including another process, receives :class:`WakeWordInUse`.
"""
if owner is None:
raise ValueError("wake-word owner must not be None")
global _detector, _detector_owner, _detector_file_lock
with _detector_lock:
if _detector is not None:
if _detector_owner is not owner:
raise WakeWordInUse("Wake-word microphone is already owned.")
_detector.on_wake = on_wake
_detector.resume()
return _detector
lock_handle = _acquire_machine_lock()
try:
cfg = config if config is not None else load_wake_word_config()
engine = _build_engine(cfg)
detector = WakeWordDetector(engine, on_wake, on_failure=_detector_failed)
_detector = detector
_detector_owner = owner
_detector_file_lock = lock_handle
detector.start()
return detector
except Exception:
if _detector is not None:
try:
_detector.stop()
except Exception:
pass
_detector = None
_detector_owner = None
_detector_file_lock = None
_release_machine_lock(lock_handle)
raise
def owns_listener(owner: object) -> bool:
with _detector_lock:
return _detector is not None and _detector_owner is owner
def pause_listening(*, owner: object) -> bool:
"""Release the microphone only when ``owner`` holds the lease."""
with _detector_lock:
if _detector is None or _detector_owner is not owner:
return False
_detector.pause()
return True
def resume_listening(*, owner: object) -> bool:
"""Re-open the microphone only when ``owner`` holds the lease."""
with _detector_lock:
if _detector is None or _detector_owner is not owner:
return False
_detector.resume()
return True
def stop_listening(*, owner: object) -> bool:
"""Fully stop the detector only when ``owner`` holds the lease."""
global _detector, _detector_owner, _detector_file_lock
with _detector_lock:
if _detector is None or _detector_owner is not owner:
return False
det = _detector
lock_handle = _detector_file_lock
_detector = None
_detector_owner = None
_detector_file_lock = None
try:
det.stop()
finally:
_release_machine_lock(lock_handle)
return True
def is_listening() -> bool:
with _detector_lock:
det = _detector
return det is not None and det.running
def audio_is_silent() -> bool:
"""True when the armed stream has delivered only silence (dead mic).
The macOS no-permission failure mode: the stream opens fine but every
frame is zeros, so detection can never fire. Lets status surfaces show
"listening but the microphone appears silent" instead of a healthy state.
"""
with _detector_lock:
det = _detector
return det is not None and det.audio_silent
def get_last_match() -> Optional[tuple[str, str]]:
"""(matched phrase, profile) of the most recent wake fire, if the engine
reports per-phrase matches (sherpa multi-profile routing). None otherwise."""
with _detector_lock:
det = _detector
if det is None:
return None
return getattr(det.engine, "last_match", None)