The voice-interruption fix (5081551f0) covered the Python surfaces (CLI +
TUI gateway) but the desktop app has its own mic path in voice-barge-in.ts,
which still had both bugs on Windows:
- HALF-DUPLEX GAP: the barge monitor only opened when TTS playback started;
during LLM generation no mic was listening at all.
- PLAYBACK DEAFNESS: the monitor calibrated its noise floor while the
speakers were already playing TTS, baking bleed into the floor. On
Windows, Chromium echoCancellation does not reliably cancel same-app
output (measured live: quiet floor ~35-50 RMS vs playback bleed
600-1700 RMS), making the trigger unreachable.
Changes mirror tools/voice_mode.full_duplex_listen:
- voice-barge-in.ts: phase-aware full-duplex monitor — quiet-only
calibration (floor held through playback, never recalibrated against
bleed), playback min-trigger clamp + ceiling, 500ms grace on playback
onset only, windowed-majority detection.
- use-voice-conversation.ts: monitor arms at turn submit and spans
generation + playback (ensureBargeMonitor, idempotent). Mid-generation
speech fires the new onInterrupt callback; spoken stop-word during a
barge ends the conversation; submit waits for the interrupt to settle.
- use-composer-voice.ts + composer/index.tsx: plumb onInterrupt: haltRun
(same seam as the Stop button).
Tests: use-voice-conversation.test.tsx (6 tests) covering monitor lifecycle
across generation + playback, mid-generation interrupt, stop-word handling.
npm run typecheck + eslint clean; remaining desktop vitest failures
reproduce on a clean tree (pre-existing Windows env failures).
Long thinking/tool stretches in a voice conversation are dead air — the
user cannot tell whether the agent is alive. New: quiet, repeating soft
bubble blips while the agent works and no speech audio is flowing.
- tools/voice_mode.py: numpy-synthesized blips (no binary assets) — two
alternating low pitches (G4/E4) with pitch glide + smooth attack/decay
envelopes, ~0.8-1.2s randomized spacing, volume = voice.beep_volume * 0.5.
start_thinking_sound(should_play=...) / stop_thinking_sound() daemon-loop
lifecycle; macOS-TCC-safe (sounddevice output gated there → silent skip,
no per-second afplay churn). New mark_audio_output_active()/
is_audio_output_active() ref-count wraps play_audio_file and the
streaming OutputStream sentence writes so "audio is flowing" is accurate.
- Config: voice.thinking_sound (default true) off-switch.
- cli.py: starts when a voice-mode turn begins, per-blip gate skips while
TTS speaks / mic records / barge capture owns the mic; stopped in the
chat() finally on every exit path.
- tui_gateway/server.py: same lifecycle around _run_prompt_submit turns
(voice mode on), gated on is_audio_output_active + continuous capture.
- Desktop: renderer owns voice-conversation audio, so a matching WebAudio
implementation (src/lib/thinking-sound.ts, same envelope/pitches) runs
while conversation status === "thinking"; honors voice.thinking_sound
(via config store) and the shared sound-mute toggle; stops instantly on
speaking/listening/end.
Saying OR typing a configured stop phrase (voice.stop_phrases, default
"stop") now ends the voice chat everywhere, not just classic CLI PTT:
- hermes_cli/voice.py: new explicit on_stop_phrase callback through
start_continuous/stop_continuous. The force-transcribe path previously
DISCARDED the stop phrase silently — with auto_restart=False the client
re-arms the next capture, so the conversation never ended. Both halt
paths now fire on_stop_phrase (fallback: on_silent_limit for legacy
callers) as user intent, distinct from the no-speech timeout.
- tui_gateway/server.py: voice.record wires on_stop_phrase and emits
voice.transcript {stop_phrase: true} after flipping HERMES_VOICE(_TTS)
off and stopping streaming TTS — same teardown as /voice off. The TTS
barge-in monitor stop-checks its transcript too. prompt.submit consumes
a TYPED bare stop phrase at the server-side choke point when voice mode
is on (returns {voice_stopped: true}, no turn starts).
- ui-tui: voice.transcript {stop_phrase} ends voice mode with a clear
'voice chat ended' notice (distinct from the no-speech-limit message);
submitPrompt releases the busy latch on a consumed voice_stopped reply.
- cli.py: _typed_voice_stop in process_loop — typing a bare stop phrase
while voice mode/continuous is active ends voice mode instead of
sending 'stop' to the agent; typed 'stop' outside voice mode is
unchanged. Voice transcripts skip the check (already stop-checked).
- desktop: interceptsTypedVoiceStop — the composer's onSubmit ends the
live voice conversation (same path as clicking end on the pill) when a
bare stop command is typed with no attachments; renderer-owned loop, so
handled client-side like the existing spoken isVoiceStopCommand.
- tools/voice_mode.py: transcribe_recording never lets the Whisper
hallucination filter swallow a configured stop phrase (e.g. 'bye'
configured as a stop phrase is both a hallucination-blocklist entry and
a stop phrase — stop-phrase check now wins).
Tests: continuous-loop signal (sabotage-verified), force-transcribe stop
signal + legacy fallback, hallucination-filter ordering, typed-stop CLI
unit tests (voice on/off/longer text), prompt.submit typed-stop gateway
tests, TUI vitest for stop_phrase event handling, desktop vitest for the
typed-stop interceptor.
The statusbar derived its client and backend version labels inline, in two
near-identical blocks. Move the wording into a pure resolver so every surface
that names an install agrees on the label, the commit diff, and the tooltip.
Assert isMessagingSource('photon') is true and that Photon/iMessage search aliases resolve, so a silent removal of the photon entry from MESSAGING_SESSION_SOURCE_IDS fails CI instead of regressing the sidebar section added in #47395.
Register Photon as a messaging platform in the Desktop sidebar:
- Add Photon brand icon (three-bar mark) to PlatformAvatar catalog
- Add photon to SOURCE_LABELS, SOURCE_ALIASES, and MESSAGING_SESSION_SOURCE_IDS
Closes#46761
xterm paints to a canvas, so its selection is not a DOM selection: the Edit
menu's Copy and the right-click Copy both call webContents.copy(), find
nothing, and copy nothing. On macOS the menu also swallows the Cmd+C
accelerator before the renderer sees it. That left Ctrl+C as the only key that
did anything, and xterm correctly forwards it to the PTY as SIGINT.
Mirror the selection into xterm's hidden helper textarea (the mechanism xterm
already uses for Linux middle-click paste) so the OS sees a real selection and
every platform copy path works, and add explicit chords on top: Cmd+C/Cmd+V on
macOS, Ctrl+Shift+C/V elsewhere, matching VS Code. Plain Ctrl+C copies only
when text is selected -- the behavior Windows Terminal and Tabby ship -- and
stays SIGINT otherwise, so interrupting a process never breaks.
Reads go through a new hermes:readClipboard IPC handler for the same reason
writes already do: the renderer's clipboard API throws whenever the document
isn't focused.
The `/` popover listed skills alphabetically, so on a 200-skill install
the ones invoked daily sat below a wall of skills that shipped with
Hermes and were never opened — /research-paper-writing (never used)
outranked /research (60 invocations).
Sort the Skills section most-used first, A-Z within a tie, and on a bare
`/` drop bundled skills with no recorded activity. Typing a query keeps
every match: a search that hides a result is broken, so a typed `/re`
only reorders.
Diagnosed from a Windows desktop's logs: the backend TTS was generating
and saving the reply mp3 every turn (provider: openai), so synthesis was
fine — the DESKTOP wasn't playing the first reply. Root cause is Chromium's
autoplay policy: audio (HTMLAudioElement.play() and AudioContext) is
suspended until the frame sees a user gesture. A voice conversation started
by the 'Hey Hermes' wake word has no preceding click, so the first reply's
play() was rejected with NotAllowedError (silently swallowed) and only
turn 2+ spoke. Manual voice-start worked because the button click WAS the
gesture — exactly the 'first message in a new voice session is silent,
manual start is fine' report.
Fix (defense in depth):
- chatWindowWebPreferences sets autoplayPolicy: 'no-user-gesture-required'
on every chat window (primary + secondary), so a deliberately-launched
native app never gates audio on a gesture. Primary fix.
- voice-playback.ts resumes a suspended AudioContext on stream 'start' and
retries HTMLAudioElement.play() once via an unlock context after a
NotAllowedError — covers the dashboard-embedded surface that doesn't get
the Electron policy.
Tests: 16 electron session-window tests (incl. new autoplay assertion),
tsc + eslint clean.
The /api/audio/speak-stream WebSocket URL was always minted for the primary
backend with no profile param, so streaming read-aloud used the default
profile's TTS provider even when another profile was active. Mint the
ticket for the active profile's connection and carry ?profile= (same seam
as /api/pty), via a read-only getApiRequestProfile() accessor.
Saying 'stop' in a voice chat did nothing — the transcript was just
submitted to the agent as a normal turn, so the conversation never ended.
The only way out was the mouse/hotkey. That's not how a hands-free voice
assistant should work.
useVoiceConversation now checks each finished utterance against a spoken
stop-command matcher BEFORE submitting: 'stop', 'stop listening', 'never
mind', 'goodbye', 'cancel', 'that's all', etc., optionally addressed
('hey hermes, stop'). A match ends the conversation (flips enabled=false,
which drives the existing end() teardown — mic close, playback stop, wake
re-arm) instead of sending a turn.
Deliberately conservative: only a WHOLE-utterance stop phrase matches, so
substantive requests that merely contain 'stop' ('stop the docker
container', 'how do I stop a process') still go through.
- apps/desktop/src/lib/voice-stop-word.ts: isVoiceStopCommand() matcher
- use-voice-conversation.ts: onStopWord option + intercept before submit
- use-composer-voice.ts: wire onStopWord -> end the conversation
- 6 matcher tests (bare/multi-word/addressed stop; substantive requests
with 'stop' pass through; bare address words don't match)
- docs: note the spoken-stop behavior
A short, bright, rising two-note ding (G5 -> C6) plays the moment 'Hey
Hermes' is detected, before voice capture starts, so it's obvious the
wake registered. Deliberately distinct from the turn-end completion cue
(that one settles; this one rises = 'listening'). Reuses the same
lightweight WebAudio synthesis as completion-sound.ts — no asset to ship
— and is gated by the shared sound-mute toggle ($hapticsMuted), so
muting turn-end sounds silences it too.
- apps/desktop/src/lib/wake-sound.ts: playWakeSound()
- wiring.tsx: fire it at the top of the wake.detected handler
- 3 tests (plays two-note chime, silent when muted, never throws with
no WebAudio)
Ear icon next to the voice controls: highlighted while listening, muted
when off, hidden when the backend reports the wake word unavailable.
Backed by a feature-owned $wakeWord nanostore synced by both the button
(wake.start/stop) and the gateway-ready auto-arm (status-then-arm), so
the UI always reflects the real listener state; start refusals surface
their reason as a tooltip notice. i18n en/zh/zh-hant/ja.
The bubble, the queue panel, and the queue editor all showed a queued or
sent /skill turn's expanded body. Thread the invocation through submit and
the queue entry, and chip a leading slash command the way a mid-prose one
already chips.
The composer re-asked the gateway for the command catalog on every open
and for complete.slash on every keystroke. Both scan the skills dir on
the backend, so opening the menu cost a round trip against data that
only moves when a skill is added, removed, or toggled.
Hold both behind a one-hour cache keyed per query, and invalidate it
where the skill set actually changes: hub install/uninstall/update, a
Capabilities toggle, bulk toggle, archive, the agent's own skill_manage
call, and a profile switch. A cached query also skips the debounce and
the spinner, so a warm menu opens in the same frame.
The bare-slash list showed no skills at all: the backend categorizes
registry commands but appends skills to the flat pairs list only, and
the popover prefers the categorized layout. Re-add the uncategorized
leftovers under a Skills header, so /clean and /work are visible on /
and not just after typing enough of the name to match.
Picking a model closed the menu and then flashed the pill's tooltip over
the fresh selection. The existing focus guard gated on `:focus-visible`
alone, which does not separate a mouse pick from a Tab: the dropdown
autofocuses its search field and keyboard-navigates its rows, so Chromium
is already in keyboard modality when the menu restores focus to the
trigger, and the guard never fired.
Track the device behind the last real interaction and use it to qualify
`:focus-visible`. A mouse pick reports `pointer` and stays silent; Tab
focus still opens the tip.
A repeated id in the transcript reached assistant-ui's MessageRepository,
which throws on the second link and takes the whole workspace pane down to
the contribution error boundary — the pane the user is actually working in.
Dedupe where the repository export is built, so no upstream transcript bug
can crash the pane, and close the journal merge path that produced one: a
resume that replays a still-journaled turn appended rows the base already
held by id.
A mid-stream steer persists an interrupted-turn checkpoint so the model knows
its reply was cut off. That scaffolding — "[This response was interrupted by a
user correction.]" and the "Visible response before the interruption:" header —
was written straight into message content, so every reload painted the raw
machinery as an assistant bubble (and merged it into the preceding tool-call
bubble). Steered transcripts became unreadable.
Reuse the existing display/replay split instead of inventing new surface:
- Carry the scaffolded form in the server-only api_content sidecar (the exact
bytes replayed to the provider), keep content the user's/agent's real words.
- When nothing reached the screen there is no clean form, so mark the row
display_kind=hidden — replayed to the model, dropped by every transcript
surface, exactly like compaction-reference rows.
- Honor display_kind=hidden in the gateway's _history_to_messages projection
(it only sniffed the [System: convention), so the checkpoint can't leak
through the live/resume path to the TUI/CLI either.
The model still receives the full interrupted context on the wire; the
transcript shows the partial reply and the user's correction.
displayContentForMessage re-derived every ref from the attached context
block and re-emitted it as a detached list above the text. Now that the
token survives expansion, that list duplicated the inline chip. Hoist
only the refs the prose is missing, so turns persisted by an older
backend still render their chips.
Empty Enter while busy was a hard no-op, so a queued turn could only be
sent early via the panel's send arrow or Cmd+Shift+K. With prompts
queued, a second Enter now promotes the head and interrupts, mirroring
the idle empty-Enter drain. Nothing queued keeps the old no-op.
The panel's send-now row action switches from a bare up-arrow to the
return glyph, so the row states the keybind the double-send uses.
The resting chip fill and the bold weight both fought the brand mark for
attention. Links now sit in the theme's primary color at body weight, and
the tint fades in on hover.
Weight is overridden on .link-chip itself: `@tailwindcss/typography` sets
`prose a { font-weight: 500 }`, which outranks a utility class on the anchor,
so the per-call-site classes could never win.
PrettyLink leads with the site's mark, so a GitHub PR link reads as a GitHub
link at a glance. The artifacts pane uses the same lookup in place of its
generic chain glyph; unknown hosts render as before.
A lookup table of ~170 hosts plus a resolver that walks the hostname's
suffixes, so subdomains inherit their parent's mark (gist.github.com) while
a more specific entry (docs.google.com) still wins over the general one.
Typing a real message instead of picking an option left the agent parked:
the clarify blocks inside its tool batch waiting on clarify.respond, so a
follow-up routed through steer/queue sat undelivered until the 5-minute
clarify timeout. skipClarifyRequest answers the parked question with the
same empty answer the card's own Skip button sends, so the tool returns
and the turn carries on with the user's words.
Letter chords now follow `event.key`, but punctuation was still anchored
to the physical QWERTY position, so the shipped punctuation defaults stayed
unreachable on a remapped layout. On Dvorak `mod+.` (command center) reads
the physical V key, and `mod+,` / `mod+/` land on `w` / `[`.
Take `event.key` for unshifted punctuation too. Shift stays excluded because
a shifted `event.key` is the shifted glyph ("?" for "/") and combos are
anchored to the unshifted token. Digits stay physical: AZERTY types "&" on
the unshifted "1" key, so `event.code` is what keeps `mod+1` bound. Glyphs
we do not ship as tokens — Option output, dead keys, non-Latin scripts —
fail both checks and fall back to the physical code.
The punctuation set derives from CODE_TO_KEY so the two cannot drift.
Two identical tool rows, one above the other, rendered at two different
opacities — and no amount of hovering would even them out.
The enter animation fills forwards, so its final keyframe is held in the
animation origin of the cascade for as long as the element lives, above the
author stylesheet. Naming `opacity: 1` there didn't just end the fade, it
permanently overruled the resting opacity of every element the sheet dims.
Transcript scaffolding is dimmed exactly that way, so a row kept whichever
opacity it happened to mount with: full if it animated in during the turn,
faded if it was rehydrated or remounted past its one-shot key. Same for
thinking headers, which is why "Thought" never matched the rows near it.
Leave the end opacity out of the keyframe. It animates up to whatever CSS
asks for and keeps answering to it, hover included.
Then close the way the surfaces drifted in the first place: the fade named
each one in its own selector, so the live status line — added later, and
neither tool nor thinking nor prose — matched none of them and sat a shade
brighter than the rows either side. One `data-conversation-scaffold` mark
now carries it, and every surface opts in.
Adds the grammar behind "Edited wiring.tsx, explored 3 files, ran 5
commands": one clause per category of work, a name when the category
holds a single thing and a count otherwise, and the present tense for
whichever category is still running.
The continuity test is the load-bearing part. Tool grouping was reverted
once because it reshuffled the moment a turn settled, so this replays the
same turn twice — as the gateway event stream the live view builds
bubbles from, and as the rows toChatMessages rehydrates on resume — and
asserts both produce the same runs.
/resume was classified as an options command, but its argument is a
free-text query the picker fuzzy-matches against session titles and
previews. Its completion list also always ends in a "Browse all
sessions…" action row, so Space-to-accept never fell through to an
empty list: the first space in a multi-word query emptied the composer
and threw the user into the overlay, query and all.
Classify it as mixed so spaces type through, matches keep narrowing as
you refine, and Tab or arrow-then-Enter still accept a highlighted
session.
The bar is always-on chrome today. Hiding it is `⌘⇧S`, the ⌘K palette, or
the bottom row of its own right-click menu — VS Code's set of doors, minus
their unbound default (they ship `toggleStatusbarVisibility` with no
keybinding and Hermes has no chord dispatcher for a `⌘K ⌘S` two-stroke).
Hidden unmounts the bar rather than hiding it, so the 15s status poll and
the per-turn readouts stop with it. Visibility persists per window profile
and defaults on.
