* feat(cli): plan catalog on Free + plan= deep link + top-up/auto-refill copy split
Bring the plain (non-TUI) CLI billing surface to parity with the desktop/TUI
billing changes:
- /subscription on Free (admin/owner, interactive) prints the plan catalog
(name · $/mo · $credits/mo, from the same tiers[] data the TUI uses; monthly
credits render as dollars). A numbered pick opens the manage-subscription
deep-link directly with plan=<tier_id> appended.
- subscription_manage_url(state, tier_id=...) appends plan=<tier_id> (the stable
tiers[] id) when a tier was picked, org_id first — mirrors the TUI's ?plan=.
The paid change flow's blocked/unknown-preview portal fallback carries plan=
for upgrades only; downgrades stay generic/native.
- /topup overview splits one-time top-up from automatic refill, the distinction
stated in each first sentence ("Add funds now — a single charge…" vs "Refill
when low — charges … automatically …"), keeping "credits" out of the
dollars-only surface.
- Downgrades remain native (chargeless scheduled change), unchanged.
Updates the CLI-parity section of docs/billing-lifecycle.md and tests under
tests/hermes_cli + tests/agent.
* refactor(billing): share plan-catalog helpers + harden manage-url builder
- subscription_manage_url now preserves unrelated portal query params (parse_qsl,
popping only the contract-owned org_id/plan) and restricts to http/https schemes,
matching the desktop URL builder — the function owns the contract.
- Lift the plan-catalog derivation into agent/subscription_view.py so the CLI Free
catalog and the paid picker/blocked-preview branch share one implementation:
selectable_tiers (enabled paid, not current, sorted), format_tier_row (name · $/mo
· $credits/mo — thousands-grouped like the TUI's toLocaleString, credits suffix
hidden when absent/zero), and is_upgrade(state, tier_id).
* fix(cli): numbered pick, canonical guarded browser opener, partial auto-refill copy
- Free catalog: accept a bare digit as a pick (the shared normalizer only knows the
confirm-dialog digit aliases, so `1` used to resolve to None → "Cancelled"). The
Nth digit maps to the Nth printed row.
- Extract one _open_url_in_browser used by every "open the portal" path, applying the
device-code flows' console-browser / remote-session guard (webbrowser.open returns
True even for lynx/w3m over SSH) and returning whether a real browser opened.
- Consume the shared selectable_tiers / format_tier_row / is_upgrade helpers from the
Free catalog, the paid picker, and the blocked-preview branch.
- /topup auto-refill copy: the concrete "charges $X … below $Y." sentence only when
both amounts are present and finite; otherwise the generic sentence.
* docs(billing): correct CLI-parity rows (drop cross-repo ref, downgrade invariant)
Remove the other-repo PR reference from the manage-URL row, and state the real
downgrade invariant: a blocked downgrade may print the generic manage URL but never
carries plan=<tier_id> — selected-tier deep-links are reserved for new subscriptions
and upgrades.
Complete the #65919 warm/live-payload fix across its sibling path and add
real-SessionDB cross-builder coverage.
- Child-watch (lazy) resume: the delegated-subagent watch window served
_history_to_messages(repaired_history) for its user-visible messages, which
collapses out persisted verification candidates just like the warm-payload
path did. Build the visible messages from the verbatim child-only display
projection (repair_alternation=False) while the repaired history still feeds
live replay; fall back to the repaired history if the display read fails.
- E2E cross-builder consistency (real SessionDB, not mocks): a persisted
verification candidate is collapsed out of the model projection but kept in
the display projection, and _live_visible_history now equals the eager
session.resume display projection (candidate present). Adds the combined
candidate + fully-flushed-second-turn case and a lazy child-watch handler
test that asserts the candidate survives in resp["result"]["messages"].
#65919 persists verification candidates (finish_reason=verification_required
/ verify_hook_continue) to state.db but collapses them out of the in-memory
model history via repair_message_sequence. The eager session.resume + REST
paths read the verbatim display lineage (candidate present), but the
warm/live-reuse payload (_live_session_payload) built its user-visible
messages from the collapsed in-memory model history — so switching to a
still-live session dropped the substantive verification answer that a cold
resume of the SAME session showed. That divergence is the cross-session
"substantive text vanishes on switch" class, and the direct sibling of the
resume-duplication regression fixed in #68149.
Reconcile the persisted display lineage (candidate-inclusive, the same
get_messages_as_conversation(..., include_ancestors=True) read the eager
resume + REST paths use) with the fresh in-memory tail in
_live_visible_history, so all three surfaces agree by construction while a
not-yet-flushed live turn is still shown. Extracted
_reconcile_display_with_live as a pure, DI-testable function (anchors on the
last persisted row's (role, text); appends only the uncovered in-memory tail;
trusts the DB display when the tail can't be anchored).
Tests: unit coverage for candidate-inclusion, freshness, empty/raising-DB
fallback, and the combined candidate+fresh-tail case. The existing freshness
guard (test_session_resume_live_payload_uses_current_history_with_ancestors)
stays green.
Changing a single color kept wrecking the rest because the agent hand-authored a
new skin (often from `default`, which has no `background`, resetting the terminal
to black). Add `hermes skin set <key> <hex>`: edits the ACTIVE skin's one key in
place (a built-in is forked into an editable copy carrying its full palette), so
everything else — background included — is preserved. Plus `skin use` / `skin
list`. The skill now points tweaks at this command instead of hand-authoring.
A skin Hermes activates (`hermes config set display.skin X`) or recolors in
place now goes live on every surface (CLI, TUI, desktop) within ~half a
second, on its own — no `/skin`, no tool-hook timing, no user action.
A gateway daemon polls the resolved skin signature `(name, active-file mtime)`
every 0.5s and broadcasts `skin.changed` on any real move — a name switch OR a
live color edit to the active skin. It routes through the SAME path `/skin`
uses, so all surfaces repaint identically. The watcher seeds its baseline at
gateway.ready (stdio + ws) so it only fires on a real change; the `/skin` RPC
seeds the baseline too so it never double-broadcasts.
Subsumes the desktop's post-turn `config.get skin` poll (its skin.changed
handler already applies).
Review on #20379, finding 1 (High). Two ways an MCP config revision could
be silently acknowledged without ever being applied:
Client: the poll advanced its accepted mcp_rev BEFORE calling reload.mcp,
and quietRpc collapses failures to null — a reload that failed against a
temporarily broken server left the revision recorded as applied, and no
subsequent poll retried until an unrelated MCP edit. The handshake is now
syncMcpReload(): send the observed rev with the request, advance `accepted`
only when the server answers status=reloaded (to the server's loaded_rev,
falling back to the requested rev on older gateways), and re-compare on
EVERY poll tick — decoupled from mtime — so a transient failure heals on
the next tick. An in-flight guard stops the 5s poll from stacking requests
behind a slow reload.
Server: generation-only coalescing let a follower triggered by revision B
ack against revision A's registry when the config changed under a slow
leader. The leader now re-hashes the MCP-relevant config after discovery
and repeats until stable (bounded), records _mcp_reload_loaded_rev, and a
follower coalesces only when the revision it was asked to load matches —
otherwise it re-runs the full reload itself. Responses carry loaded_rev.
Deterministic tests for the exact failure sequences: failed reload → no
ack, no generation advance; A-then-B overlap → follower re-runs; matching
rev → coalesces; failed leader → follower re-runs; legacy no-rev callers
keep generation-only coalescing (thread ordering via an instrumented lock,
no sleeps). Client: 6 vitest cases on the ack/retry/in-flight contract.
subprocess.run(["git", ...], timeout=...) deadlocks on Windows: run()'s
post-timeout cleanup calls an unbounded communicate() after killing git.
Killing the PATH-resolved launcher can leave a suspended descendant git.exe
holding duplicates of the captured stdout/stderr handles, so the pipes never
reach EOF and the reader-thread join blocks forever — leaking a process +
two reader threads per fired timeout (the accumulating git.exe load behind
Windows Defender CPU spikes).
Two fail-open probe call sites had this identical flaw:
- tui_gateway/git_probe.py::run_git — on the Desktop agent-build path
(_start_agent_build -> _session_info -> branch() -> run_git), where the
hang turned an optional branch label into "agent initialization timed
out" (#68609).
- agent/coding_context.py::_git — hangs the agent turn inside
build_coding_workspace_block under an ACP host (#66037).
Consolidate both onto one shared bounded_git_probe() in
hermes_cli/_subprocess_compat.py (both files already import from there, so
no new import surface):
- explicit communicate(timeout), then on ANY failure a tree-kill —
proc.kill() AND, on Windows, best-effort taskkill /T /F so the suspended
descendant that holds the pipe writers dies too — plus a bounded 1s
post-kill drain; if the pipes are still held they're abandoned (the
orphaned reader threads are daemonic and cost nothing).
- fail open to "" on every path: spawn error, timeout, kill() raising
(access denied / already reaped — a raise inside the except handler
previously escaped the contract), and non-timeout communicate() failures
now also terminate the child instead of leaving it running.
- the taskkill spawn can't re-enter the deadlock class: it captures no
pipes (DEVNULL), so its own timeout cleanup has no reader threads to join.
Normal-path spawn contract is preserved byte-for-byte: PIPE/PIPE/DEVNULL,
text + utf-8 errors="replace", hidden-window creationflags on Windows only,
nonzero returncode -> "". Each call site keeps its own timeout (1.5s / 2.5s).
Supersedes #68622 (Sora-bluesky — git_probe fix + tree-kill) and #66038
(iamwongeeeee — coding_context fix), folding both into one shared helper so
the two sites can't drift and every timeout tree-kills the descendant. Tests
consolidated onto the helper, incl. the previously-missing assertion that a
Windows timeout escalates to taskkill /T /F.
Co-authored-by: Sora-bluesky <sora.bluesky.dev@gmail.com>
Co-authored-by: iamwongeeeee <wykim777@naver.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
test_session_expired_retry_waits_for_new_session hardcoded
_server_breaker_opened_at["hindsight"] = 123.0 to simulate a circuit breaker
whose cooldown has already elapsed. But the breaker in tools/mcp_tool.py
compares that stamp against time.monotonic() (age = monotonic() - opened_at,
elapsed when age >= _CIRCUIT_BREAKER_COOLDOWN_SEC). time.monotonic()'s origin
is arbitrary and small on a freshly-booted CI container, so age worked out to
only a few seconds there (< the 60s cooldown) — the breaker stayed open, the
half-open probe never fired, and the retry returned the "unreachable" error
instead of "bank ok". It passed on long-uptime dev boxes (large monotonic)
and failed under CI, with the reported "Auto-retry available in ~Ns" drifting
run to run as the container's monotonic clock varied.
Stamp opened_at relative to the same clock the code reads
(time.monotonic() - _CIRCUIT_BREAKER_COOLDOWN_SEC - 1.0) so the cooldown is
provably elapsed regardless of the monotonic origin, exercising the intended
half-open transition deterministically.
Pixel-sampled against the reference screenshots: the beloved classic
light-mode look is the vivid authored golds rendered essentially raw
(#FFD700 shows as bright #F5C242) — transparent terminal profiles apply no
contrast lift of their own, and pre-darkening every foreground to WCAG 4.5
produced the reported mustard mud. Light-mode display floors become
near-invisible rescues only (1.18 display / 1.6 semantic; dark keeps
1.45/2.2); the default skin ships a fills-only light_colors OVERLAY
(polarity-flip the navy menu/status fills, foregrounds inherit the vivid
colors), and themeForSkin merges polarity overlays instead of replacing.
Palette audit reworked: base colors audited fully, overlays for valid keys
and fill polarity.
The desktop color-mix system, ported: a theme is a handful of identity
SEEDS (text, primary, accent, border, status hues); every secondary tone
(muted, label, surfaces, chips, selection) is a color-mix derivative
against the real terminal background. lib/color.ts consolidates the
primitives (parse/mix/luminance/contrast/retone + xterm.js's multiplicative
liftForContrast). Knobs are grid-search fitted so the math reproduces the
classic hand-tuned literals (contract-tested). The display shim renders
authored palettes RAW and only rescues near-invisible colors. Boot reads
the last resolved theme from $HERMES_HOME/tui-theme-boot.json so the first
frame paints in the right palette (no default-dark flash), and the
placeholder cursor follows the bubbles textinput pattern.
OSC-11 asks the terminal for its actual background at startup (env
heuristics are blind on xterm.js hosts) and the theme re-derives against
the answer: desktop-contract adaptation (contrast floors + fill polarity),
a shared list-row selection primitive instead of per-picker panel fills,
paired light_colors/dark_colors skin blocks with a machine audit, and a
/theme auto|light|dark pin (display.tui_theme) for hosts whose probe lies.
E2E coverage for the OSC reply chain + /theme-info diagnostics.
The /api/status gateway_updated_at field and the gateway /health/detailed
updated_at field passed through whatever gateway_state.json contained,
untyped. All current writers emit RFC3339 via _utc_now_iso(), but legacy
gateways wrote unix epoch floats, and a corrupt or hand-edited state file
can inject numbers or arbitrary garbage — while the frontend types
(web/src/lib/api.ts) declare string | null.
Add normalize_updated_at() in gateway/status.py as the single funnel:
- str: accepted iff datetime.fromisoformat parses (trailing Z tolerated);
naive timestamps coerced to UTC; canonical isoformat returned
- int/float: treated as unix epoch seconds -> UTC ISO string, with a
plausibility guard (reject < 2000-01-01, > now+1day, non-finite)
- bool: rejected explicitly (int subclass, but never a timestamp)
- anything else: None
Apply it at both emit sites: the dashboard /api/status handler (covers
both the local read_runtime_status() branch and the remote
/health/detailed cross-container fallback branch) and the gateway API
server's /health/detailed response.
Contract tests: parametrized /api/status normalization (epoch float/int,
garbage string, None, bool, dict, absent key), remote-health numeric and
garbage bodies, dashboard shape test round-trip assertion, direct
normalize_updated_at units (range guards, Z suffix, naive coercion,
non-finite floats), and a write_runtime_status -> read_runtime_status
round-trip proving the writer side stays tz-aware parseable.
The dashboard's own liveness surface could report healthy while every
authenticated request 500'd (e.g. wedged state DB) — /api/status carried
gateway-only fields, no storage/dashboard signal, and no middleware
counted unhandled exceptions.
- DashboardHealth state holder: rolling 5-min deque of unhandled-error
timestamps + last self-test result. last_error_type/last_error_path
are internal-only diagnostics — snapshot() exports counts/enums/
timestamps exclusively (PUBLIC_API_PATHS no-secrets contract).
- Outermost @app.middleware('http') (registered last) wraps call_next
in try/except: records + re-raises unhandled exceptions, and records
responses with status >= 500.
- /api/status gains 'components' {gateway, storage, dashboard,
platforms} + top-level 'overall' ok|degraded. storage reuses the
gateway readiness state_db probe (read-only, 1s-bounded) in an
executor; platforms derive ok/degraded from existing
gateway_platforms states.
- Authenticated self-test task started in the lifespan: every 60s an
in-process httpx ASGITransport GET of /api/sessions?limit=1 with the
real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly
when httpx is unavailable and while the OAuth gate is engaged (the
legacy token is not honoured there).
Tests: middleware increments on raising route and on 5xx, window
expiry, components shape + overall, storage degraded when the state_db
probe fails, dashboard degraded after an error, no secret-bearing
fields in the public payload, self-test pass/fail recording (mocked
client) + a real ASGI round trip.
_readiness_work_counts()'s active_api_runs set is {"queued", "running",
"waiting_for_approval"} — it excludes "stopping", the status
_handle_stop_run() sets while a run is being interrupted. Since the stop
is fully cooperative (the run stays "stopping" — doing real
executor-thread work — until the agent actually notices the interrupt and
the task settles to "cancelled", an unbounded window, not a fixed
timeout), /health/detailed's background_queues.active_api_runs
undercounts real active work for that whole duration.
Fix: add "stopping" to the active-status set. background_queues.status
itself is hardcoded "ok" (gateway/readiness.py), so this doesn't change
overall readiness — it only corrects the count value external monitoring
tooling reads from this endpoint.
/health and /health/detailed resolve the version via importlib.metadata
first, falling back to hermes_cli.__version__. On editable/source
checkouts — including the standard git-based install that hermes-setup
performs — hermes_agent-*.dist-info can survive a source update
unchanged, so the health endpoints keep reporting the previous release
even though the running code (CLI, dashboard, release tags) is newer.
Stale metadata does not raise, so the source fallback never fires.
Flip the preference: use hermes_cli.__version__ (the runtime source of
truth shared by the CLI and dashboard) first, and fall back to
distribution metadata only when the source import fails. The
never-raise contract of the version probe is unchanged.
Observed live: CLI, dashboard, and pyproject all reported 0.18.2 while
/health returned 0.18.0 from a stale hermes_agent-0.18.0.dist-info left
behind by a source update.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TUI retry-allowance follow-up made tui_gateway.entry.wait_for_mcp_discovery
delegate to hermes_cli.mcp_startup unconditionally when no entry-local thread
exists. But server._make_agent already calls the startup wait directly for
dashboard /api/ws sessions, so every non-stdio agent build paid the bounded
wait twice (caught by test_make_agent_waits_for_shared_mcp_discovery). Gate
the retry-spawn AND the delegated wait on _mcp_discovery_enabled, which only
the stdio TUI arms in main().
Builds on fazerluga-creator's #66981 (cherry-picked as the previous two
commits). start_background_mcp_discovery()'s retry allowance only fires
when the function is CALLED again, but tui_gateway/entry.py main() calls
it exactly once at startup — so a first discovery run that connected
nothing still latched the stdio TUI MCP-less for the whole session.
Re-invoke the idempotent spawn from wait_for_mcp_discovery() (the
per-agent-build wait) when the process is MCP-enabled, gated on a flag
set in main() so non-MCP sessions never pay the MCP import on the wait
path. Adds regression tests for both the retry re-invocation and the
non-MCP skip.
Review follow-up: the stdio hermes --tui path spawned its own one-shot
discovery thread, so the retry-after-zero-connected semantics added to
start_background_mcp_discovery() did not cover it.
Spawn TUI discovery through the shared owner and make the entry-side
wait_for_mcp_discovery() fall through to the shared owner when no local
thread exists (mcp_discovery_in_flight/join_mcp_discovery already consult
both owners). Keeps the cheap no-mcp-servers config guard on the TUI path.
Adds regression tests for the entry-side wait delegation.
start_background_mcp_discovery() sets _mcp_discovery_started once and never
resets it. If the first background run exits without connecting any MCP
server (startup cancellation, OOM restart, transient network failure), every
later call returns immediately and the process is permanently stuck with
zero MCP tools until a full restart.
Fix: when discovery is marked started but the thread is dead and no server
is connected, reset the flag and spawn a fresh discovery thread. Also log a
WARNING when a discovery run completes with zero connected servers, so the
condition is visible instead of silent.
Caught in production on a long-running gateway fleet where a gateway
restarted under memory pressure and came back with all MCP tools missing.
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.
Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
Builds on trevorgordon981's #50589 (cherry-picked as the previous commit).
The #50394 cooldown reset only ran inside the async _shutdown coroutine,
which is skipped on the empty-_servers fast path — the most common state
when a server failed to connect (failed servers are never recorded in
_servers). It was also skipped when the MCP loop wasn't running.
Clear _server_connect_retry_after/_server_connect_failures on the fast
path and in a final unconditional sweep so a full shutdown/restart always
re-attempts every configured server immediately. Adds regression tests
for both paths.
Log-storm hygiene for the reconnect machinery (#65673, #66092):
- State transitions carry exactly one WARNING each:
connected → degraded (keepalive failure), degraded → parked
(budget exhausted / rapid-drop park / permanent error),
parked → connected (revival proven healthy, via
_mark_session_proven).
- Per-attempt retry logs (initial-connect attempts, connection-lost
reconnect attempts, per-cycle rebuild notices, park-wake probes)
demoted to DEBUG.
- Backoff sleeps get +/-20% uniform jitter (_jittered) so a herd of
servers that lost the same backend doesn't retry — and log — in
lockstep.
- Add module-level _unwrap_exception_group (ported from
hermes_cli/mcp_config.py, adapted): handles nested groups, prefers
non-cancellation leaves over the CancelledErrors anyio sprays across
sibling tasks, and re-raises KeyboardInterrupt/SystemExit leaves.
- Add _classify_mcp_failure(exc) -> 'permanent'|'transient'.
Permanent: auth 401/403, NonMcpEndpointError, InvalidMcpUrlError,
FileNotFoundError/ENOENT on the stdio command. Transient: everything
else (network/EOF/ClosedResource/TaskGroup drops). Permanent failures
park immediately without burning the retry ladder — every retry
against a missing binary or a revoked credential hits the same wall.
- Every log site in run() and the keepalive failure log now logs the
unwrapped root cause as 'TypeName: message', so a dead stdio pipe
says 'BrokenPipeError' instead of the opaque
'unhandled errors in a TaskGroup (1 sub-exception)' (and empty
str(exc) dead-pipe errors are no longer blank).
Harden the immediate-reconnect path from #66271:
- _reconnect_or_reraise_group re-raises when the transport TaskGroup
carries a KeyboardInterrupt / SystemExit leaf — fatal signals must
propagate to the interpreter, never be converted into a reconnect.
- Rapid-drop budget: a completed handshake alone no longer clears
_reconnect_retries/backoff on clean transport return. A session is
UNPROVEN until it demonstrates real health — survived >=1 full
keepalive interval (keepalive success path) or served >=1 successful
tool call (_mark_session_proven). Unproven clean returns are charged
against _MAX_RECONNECT_RETRIES, so a flapping transport that
handshakes fine and drops moments later still reaches the park
instead of respawning forever (#62212: 6212 spawns in 63h).
Proven sessions keep the #57604 behaviour: budget clears, transient
blips over a long-lived session never accumulate toward parking.
Streamable-HTTP / SSE MCP transports run their stream pump inside an anyio
TaskGroup. A transient stream drop (idle timeout, brief backend blip,
server-side TCP close) surfaces as a BaseExceptionGroup escaping the
transport context manager. It reached run()'s error path, which applied
exponential backoff (1s..16s) and eventually parked the server for 300s and
deregistered its tools — turning a sub-second glitch into a multi-minute
tool outage even though the POST path was still healthy.
_run_http now wraps all three transport branches (SSE, new + deprecated
Streamable-HTTP) and routes a BaseExceptionGroup through a new
_reconnect_or_reraise_group() helper that returns 'reconnect' (immediate
rebuild, no backoff/park/deregister). It re-raises instead of masking when
shutdown is in progress, when the group carries a real CancelledError
(cancellation must propagate, cf. #9930), or when no live session was
established this attempt (a connect/handshake failure that should fall
through to run()'s backoff rather than hot-loop).
Fixes#66092
Co-authored-by: 雨哥 <hanyu1212@users.noreply.github.com>
PR #63455 shipped the flock without tests. Cover the three contention
paths: contended+dist serves stale without a second build, uncontended
builds under the lock (creating the lock file), and contended+no-dist
blocks then skips the rebuild because the staleness re-check runs under
the lock and sees the winner's stamp. Also pin the lock filename into
.gitignore via a regression assertion.
Replace mtime-based web UI dist staleness with a content-hash stamp under HERMES_HOME, matching the desktop build freshness model.
This avoids false stale/fresh decisions when git operations rewrite source mtimes without changing bytes, while preserving the existing stale-dist fallback. Also treats malformed or missing stamp data as needing a rebuild.
mount_spa degrades to a JSON 404 catch-all when the dist directory is
fully missing, but _serve_index read index.html unguarded — so a dist
dir that exists while index.html is missing (partial build, wiped dist,
permissions) raised FileNotFoundError on EVERY request instead of
returning a useful error.
Catch OSError around the read and return the same JSON 404 payload the
fully-missing-dist path uses, so clients get a consistent signal. The
route recovers automatically once a rebuild restores the file.
--skip-build with a missing web_dist/index.html previously hard-failed
with sys.exit(1) (issue #59288). The desktop launcher passes
--build-mode skip on every boot, so a wiped or never-populated dist
bricked the dashboard until the user manually rebuilt.
Now the default-dist path logs a clear warning and attempts exactly ONE
recovery build through the existing _build_web_ui path. If the recovery
build also fails to produce index.html, the original fatal behavior is
preserved with a clear message. A custom HERMES_WEB_DIST stays fail-fast:
the build writes to the default dist location and cannot populate a
caller-managed directory.
Closes#59288
Wheels ship hermes_cli/web_dist via pyproject package-data, but the sdist
did not: MANIFEST.in had no graft and .gitignore excludes web_dist, so
source tarballs installed a dashboard-less package. Graft the directory
and add an sdist regression test that builds the tarball and asserts
index.html is inside.
Salvaged from #29661; the PR's [web]-extra 404-message change was dropped
per maintainer review (misleading guidance for source installs).
Adds kanban_db.repair_db() — a structured, non-raising wrapper around
the same narrow repair policy as the connect-time guard: probe with
PRAGMA integrity_check under the board's cross-process init flock;
quarantine the corrupt bytes FIRST via the content-addressed backup;
REINDEX only when every integrity message is index-scoped; re-check;
report ok / repaired / corrupt / missing. Locked/busy OperationalError
still propagates raw (a locked healthy DB is not corruption and gets
no quarantine), and a repair invalidates the per-process healthy-path
cache so the next connect() re-probes.
The CLI verb reports status human-readably (or --json), exits 0 for
ok/repaired/missing and 1 when the DB is still corrupt (non-index
corruption stays fail-closed with manual-recovery guidance). It
dispatches BEFORE kanban_command's auto-init: init_db() raises
KanbanDbCorruptError on a corrupt board, which previously would have
made a repair verb unreachable on exactly the boards that need it.
CLI tests drive the real argparse surface (build_parser +
kanban_command) against real corrupted SQLite fixtures.
Kanban connections set wal_autocheckpoint=100, but SQLite's passive
autocheckpoint backs off whenever any reader holds an open snapshot —
on a busy multi-process board the -wal file can grow without bound
between gateway restarts.
After each successful dispatch tick, while still holding the board's
single-writer dispatch flock, run PRAGMA wal_checkpoint(TRUNCATE)
best-effort at a coarse interval (>=5 min since this process last
checkpointed that board; module-level per-path monotonic timestamp, so
multi-board dispatchers checkpoint each board on its own clock).
Success and busy/locked skips are both logged at DEBUG; a failing
checkpoint can never fail the tick.
Content-addressed quarantine backups dedupe identical corrupt bytes,
but corruption that keeps mutating between failures (partial repairs,
further damage across dispatcher retries, multi-profile fleets) mints a
new sha-named backup every round — a user accumulated 124
.corrupt.*.bak files with no bound.
After each NEW backup is created, prune oldest-by-mtime backups beyond
_CORRUPT_BACKUP_RETENTION (module constant, default 10), including the
copied -wal/-shm sidecars. The just-created backup is always exempt
(copy2 preserves the source mtime, which can be older than existing
backups). Pruning is best-effort and never masks the corruption error
about to be raised; dedupe of identical corrupt bytes is unchanged.
_guard_existing_db_is_healthy previously failed closed on ANY
integrity_check failure, including the index-scoped class ('wrong # of
entries in index <name>' / 'row N missing from index <name>') where the
table b-trees are intact and REINDEX rebuilds the damaged indexes
losslessly. Boards hit by that class were bricked until manual surgery
even though SQLite can fix them in-place.
Now, when integrity_check output consists ONLY of index-scoped errors
(index name parsed generically from the message — no hardcoded list):
1. quarantine the corrupt bytes FIRST via the existing content-
addressed _backup_corrupt_db,
2. under the caller-held cross-process init flock, REINDEX each named
index (falling back to bare REINDEX if a parsed name doesn't
resolve),
3. re-run integrity_check and proceed only if it comes back clean.
Any non-index error class (page corruption, malformed image, freelist
damage) — or a REINDEX whose re-check is still dirty — fails closed
exactly as before: backup + KanbanDbCorruptError, no silent recreation.
Transient OperationalError (locked/busy) still propagates raw with no
quarantine.
Tests build a real board DB and corrupt a live index via the
writable_schema/partial-index REINDEX trick to produce the genuine
'wrong # of entries in index' shape, then assert auto-repair recovers
with data intact, page corruption still raises, and a dirty re-check
fails closed.
Replace the mocked test for #63398's REINDEX strategy: the original
monkeypatched _db_opens_cleanly to return the corruption string, so the
REINDEX pass itself was never exercised against actual index corruption —
the test would pass even if REINDEX didn't fix anything.
New fixture _corrupt_btree_index() builds genuine on-disk staleness with a
writable_schema hack: rewrite the index definition to a partial index
(WHERE 0), REINDEX so the b-tree is rebuilt empty, then restore the full
definition. integrity_check then reports the real
'wrong # of entries in index idx_messages_session' / 'row N missing from
index' class from #63386 — no mocks anywhere.
The rewritten test asserts end-to-end with real function calls:
- the real _db_opens_cleanly detects the stale index,
- repair_state_db_schema repairs it with strategy 'reindex_btree',
- post-repair the detector and raw PRAGMA integrity_check both report
healthy, and a query forced through the rebuilt index (INDEXED BY) sees
every row.
Adds a second test asserting the REINDEX strategy is non-destructive
(all sessions/messages survive, readable via SessionDB).
Follow-up to #63398; refs #63386
When PRAGMA integrity_check reports 'wrong # of entries in index' for
B-tree indexes (e.g. idx_sessions_handoff_state), the existing repair
strategies (FTS rebuild, sqlite_master dedup, drop-FTS+VACUUM) don't
address the mismatch. Add Strategy 0.5: run REINDEX to rewrite the
index b-tree from canonical table rows before escalating to more
destructive strategies.
The trigram MATCH branch in search_messages() had the same
OperationalError-only catch that #66420 fixed on the main FTS5 branch: a
corrupt messages_fts_trigram shadow table raises the malformed /
'fts5: corrupt structure record' class (sqlite3.DatabaseError, parent of
OperationalError), which propagated straight out of search_messages and
crashed CJK session/history search for read-only sessions.
Route that class through the shared one-shot _try_runtime_fts_rebuild()
and retry the trigram query (catch moved outside self._lock so
rebuild_fts() can re-acquire it, mirroring the main branch). If the
rebuild is refused (guard consumed / FTS disabled / different error) or
the retry fails, fall through to the existing LIKE substring fallback —
which reads only the canonical messages table — instead of raising, so
CJK search degrades gracefully rather than crashing.
Adds two regression tests: trigram search self-heals in place after
shadow-table corruption (answers from the rebuilt trigram index, not the
LIKE fallback), and degrades to LIKE without raising when the one-shot
rebuild was already consumed.
Follow-up to #66420; refs #66296#66724
Complements #66296 (self-heal on the write path): search_messages()'s main
FTS5 MATCH query caught only sqlite3.OperationalError (a query-syntax error →
return empty). A corrupt FTS index raises the malformed / "fts5: corrupt
structure record" class, which is a sqlite3.DatabaseError — the parent of
OperationalError, so it was NOT caught and propagated straight out of
search_messages, crashing session/history search.
The write path now rebuilds and retries on that class, but a read-only
session (cron/CLI history search, or a search issued before any write) never
triggers a write, so its search stayed broken until the next process restart
ran the offline repair.
Catch the DatabaseError corruption class on the search MATCH read too and
route it through the existing one-shot _try_runtime_fts_rebuild(), then retry
the query. The catch is moved outside `with self._lock` so rebuild_fts() can
re-acquire the lock (mirrors _execute_write). The one-shot guard is shared
with the write path, so a single instance never loops on a genuinely
unrecoverable index. OperationalError syntax handling is unchanged (caught
first).
Adds a regression test: with a corrupted messages_fts and no post-corruption
write, search_messages() rebuilds in place and returns the match; without the
fix it raises DatabaseError.
Two follow-ups on top of f842733 (the FTS5 read probe added in #66906):
1. The original probe query used MATCH '', which FTS5 rejects with
'fts5: syntax error near '. Empty MATCH syntax is not valid FTS5.
Switch to MATCH '""' — a quoted empty phrase that parses, scans
zero rows, and exercises the same shadow-table read path the
search tools use. The probe previously never reached the shadow
segments at all on a healthy DB; the read-corruption class was
only being detected because the existing write probe happens to
fail first on a DatabaseError.
2. The probe's degraded-runtime branch only checked the substrings
'no such table' / 'no such column'. On a SQLite build without the
fts5 module, MATCH against a legacy messages_fts table raises
'no such module: fts5' (a different OperationalError class). The
substring check would misclassify that as corruption and trigger
repair, whose final fallback deletes the messages_fts% schema
(#66906 review). Use SessionDB._is_fts5_unavailable_error() — the
canonical classifier already used by the degraded-runtime init
path — to recognize both 'no such module: fts5' and
'no such tokenizer: trigram' as capability errors.
Add tests covering:
- Partial shadow-table damage (read-corruption class)
- Repair brings reads back online
- Healthy degraded DB without fts5 module stays healthy (regression
for the misclassification risk)
- Healthy degraded DB without trigram tokenizer stays healthy
Closes#66906 review feedback
Refs #66724
