Commit graph

16710 commits

Author SHA1 Message Date
Brooklyn Nicholson
cd05498e2c feat(ui-tui): background-aware theme adaptation + paired palettes + /theme pin
OSC-11 asks the terminal for its actual background at startup (env
heuristics are blind on xterm.js hosts) and the theme re-derives against
the answer: desktop-contract adaptation (contrast floors + fill polarity),
a shared list-row selection primitive instead of per-picker panel fills,
paired light_colors/dark_colors skin blocks with a machine audit, and a
/theme auto|light|dark pin (display.tui_theme) for hosts whose probe lies.
E2E coverage for the OSC reply chain + /theme-info diagnostics.
2026-07-21 16:36:57 -05:00
Brooklyn Nicholson
4a129af709 refactor(ui-tui): route production surfaces through the widget-grid engine
Banner (responsive tiers: full logo -> compact rule -> text -> hidden),
SessionPanel (fixed hero track + flexible info track, the desktop pane
shell's fixed-vs-flex contract), floating overlays, prompt zone, and the
pickers all render through WidgetGrid instead of hand-rolled flex math.
Pickers gain maxWidth so grid cells can cap them.
2026-07-21 16:36:17 -05:00
Brooklyn Nicholson
58d75ec5c6 feat(ui-tui): widget-grid 2-axis layout engine + overlay primitives (#20379 rebase)
Rebase of the widget-grid PR onto current main, then grow it into a real
2-axis engine: resolveGridTracks (grid-template tracks — fixed cell counts
and weighted fr shares with mins), layoutWidgetGrid (1D auto-packing flow),
and layoutGridAreas (2D absolute placement with row/col spans and implicit
row growth). Overlay/Dialog primitives give zoned viewport-level modals
with an optional scrim. /grid-test (interactive: areas, nesting, zoom,
gap/padding) and /grid-test streams (4x3 mission-control GridAreas board
with promote-to-main) exercise everything end to end.
2026-07-21 16:36:17 -05:00
brooklyn!
81f4095bc1
Merge pull request #68918 from NousResearch/bb/composer-focus-keys
feat(desktop): type-to-focus the composer from empty chat chrome
2026-07-21 16:33:45 -05:00
Brooklyn Nicholson
a1a406fd40 feat(desktop): type-to-focus the composer from empty chat chrome
Printable keys and soft `/`/Enter pull focus back to the composer when
nothing else owns the key (dialogs, terminal, buttons on Enter, …).
2026-07-21 16:03:19 -05:00
brooklyn!
413ed6b9df
Merge pull request #68860 from NousResearch/bb/battery-status
feat(status-bar): add /battery toggle for a color-coded battery read-out
2026-07-21 15:33:21 -05:00
Teknium
a2c2ec6332
fix(status): make gateway_updated_at a stable RFC3339-or-null contract (#68657)
The /api/status gateway_updated_at field and the gateway /health/detailed
updated_at field passed through whatever gateway_state.json contained,
untyped. All current writers emit RFC3339 via _utc_now_iso(), but legacy
gateways wrote unix epoch floats, and a corrupt or hand-edited state file
can inject numbers or arbitrary garbage — while the frontend types
(web/src/lib/api.ts) declare string | null.

Add normalize_updated_at() in gateway/status.py as the single funnel:
- str: accepted iff datetime.fromisoformat parses (trailing Z tolerated);
  naive timestamps coerced to UTC; canonical isoformat returned
- int/float: treated as unix epoch seconds -> UTC ISO string, with a
  plausibility guard (reject < 2000-01-01, > now+1day, non-finite)
- bool: rejected explicitly (int subclass, but never a timestamp)
- anything else: None

Apply it at both emit sites: the dashboard /api/status handler (covers
both the local read_runtime_status() branch and the remote
/health/detailed cross-container fallback branch) and the gateway API
server's /health/detailed response.

Contract tests: parametrized /api/status normalization (epoch float/int,
garbage string, None, bool, dict, absent key), remote-health numeric and
garbage bodies, dashboard shape test round-trip assertion, direct
normalize_updated_at units (range guards, Z suffix, naive coercion,
non-finite floats), and a write_runtime_status -> read_runtime_status
round-trip proving the writer side stays tz-aware parseable.
2026-07-21 13:12:41 -07:00
Teknium
8fd5b25898
feat(dashboard): component-level health rollup on /api/status (#68662)
The dashboard's own liveness surface could report healthy while every
authenticated request 500'd (e.g. wedged state DB) — /api/status carried
gateway-only fields, no storage/dashboard signal, and no middleware
counted unhandled exceptions.

- DashboardHealth state holder: rolling 5-min deque of unhandled-error
  timestamps + last self-test result. last_error_type/last_error_path
  are internal-only diagnostics — snapshot() exports counts/enums/
  timestamps exclusively (PUBLIC_API_PATHS no-secrets contract).
- Outermost @app.middleware('http') (registered last) wraps call_next
  in try/except: records + re-raises unhandled exceptions, and records
  responses with status >= 500.
- /api/status gains 'components' {gateway, storage, dashboard,
  platforms} + top-level 'overall' ok|degraded. storage reuses the
  gateway readiness state_db probe (read-only, 1s-bounded) in an
  executor; platforms derive ok/degraded from existing
  gateway_platforms states.
- Authenticated self-test task started in the lifespan: every 60s an
  in-process httpx ASGITransport GET of /api/sessions?limit=1 with the
  real _SESSION_TOKEN, feeding the dashboard component. Skips cleanly
  when httpx is unavailable and while the OAuth gate is engaged (the
  legacy token is not honoured there).

Tests: middleware increments on raising route and on 5xx, window
expiry, components shape + overall, storage degraded when the state_db
probe fails, dashboard degraded after an error, no secret-bearing
fields in the public payload, self-test pass/fail recording (mocked
client) + a real ASGI round trip.
2026-07-21 13:11:58 -07:00
Teknium
e2e8823f1d chore(contributors): map yingwaizhiying@gmail.com -> tianma-if (supersedes stale msh01 entry) 2026-07-21 12:56:31 -07:00
pierrenode
0d5982d910 fix(api-server): count "stopping" runs as active in readiness work counts
_readiness_work_counts()'s active_api_runs set is {"queued", "running",
"waiting_for_approval"} — it excludes "stopping", the status
_handle_stop_run() sets while a run is being interrupted. Since the stop
is fully cooperative (the run stays "stopping" — doing real
executor-thread work — until the agent actually notices the interrupt and
the task settles to "cancelled", an unbounded window, not a fixed
timeout), /health/detailed's background_queues.active_api_runs
undercounts real active work for that whole duration.

Fix: add "stopping" to the active-status set. background_queues.status
itself is hardcoded "ok" (gateway/readiness.py), so this doesn't change
overall readiness — it only corrects the count value external monitoring
tooling reads from this endpoint.
2026-07-21 12:56:31 -07:00
Kennedy Umege
eff293115d fix(gateway): report runtime source version in /health, not stale dist-info metadata
/health and /health/detailed resolve the version via importlib.metadata
first, falling back to hermes_cli.__version__. On editable/source
checkouts — including the standard git-based install that hermes-setup
performs — hermes_agent-*.dist-info can survive a source update
unchanged, so the health endpoints keep reporting the previous release
even though the running code (CLI, dashboard, release tags) is newer.
Stale metadata does not raise, so the source fallback never fires.

Flip the preference: use hermes_cli.__version__ (the runtime source of
truth shared by the CLI and dashboard) first, and fall back to
distribution metadata only when the source import fails. The
never-raise contract of the version probe is unchanged.

Observed live: CLI, dashboard, and pyproject all reported 0.18.2 while
/health returned 0.18.0 from a stale hermes_agent-0.18.0.dist-info left
behind by a source update.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 12:56:31 -07:00
yoma
87e31cb7d1 fix(dashboard): cap gateway health probe timeout 2026-07-21 12:56:31 -07:00
Brooklyn Nicholson
3c2903a2fb fix(status-bar): address Copilot review on /battery
- TUI /battery matches the CLI surface: adds `status` (live reading via
  system.battery), and the help/usage strings now consistently read
  [on|off|status].
- batteryLabel() renders `--` for an unknown percent so a null can never
  surface as "null%" even without the showBattery guard.
- Move the system.battery RPC out of the config section into
  "Methods: tools & system" where system.* RPCs belong.
2026-07-21 14:47:49 -05:00
Teknium
f218474552 fix(tui): gate the shared-owner MCP discovery wait on the stdio TUI flag
The TUI retry-allowance follow-up made tui_gateway.entry.wait_for_mcp_discovery
delegate to hermes_cli.mcp_startup unconditionally when no entry-local thread
exists. But server._make_agent already calls the startup wait directly for
dashboard /api/ws sessions, so every non-stdio agent build paid the bounded
wait twice (caught by test_make_agent_waits_for_shared_mcp_discovery). Gate
the retry-spawn AND the delegated wait on _mcp_discovery_enabled, which only
the stdio TUI arms in main().
2026-07-21 12:43:27 -07:00
Teknium
c54b568a38 fix(tui): give the stdio TUI discovery a retry-after-zero-connected path
Builds on fazerluga-creator's #66981 (cherry-picked as the previous two
commits). start_background_mcp_discovery()'s retry allowance only fires
when the function is CALLED again, but tui_gateway/entry.py main() calls
it exactly once at startup — so a first discovery run that connected
nothing still latched the stdio TUI MCP-less for the whole session.

Re-invoke the idempotent spawn from wait_for_mcp_discovery() (the
per-agent-build wait) when the process is MCP-enabled, gated on a flag
set in main() so non-MCP sessions never pay the MCP import on the wait
path. Adds regression tests for both the retry re-invocation and the
non-MCP skip.
2026-07-21 12:43:27 -07:00
fazerluga-creator
8b6e92acf5 fix(tui): centralize stdio TUI MCP discovery on the shared owner
Review follow-up: the stdio hermes --tui path spawned its own one-shot
discovery thread, so the retry-after-zero-connected semantics added to
start_background_mcp_discovery() did not cover it.

Spawn TUI discovery through the shared owner and make the entry-side
wait_for_mcp_discovery() fall through to the shared owner when no local
thread exists (mcp_discovery_in_flight/join_mcp_discovery already consult
both owners). Keeps the cheap no-mcp-servers config guard on the TUI path.

Adds regression tests for the entry-side wait delegation.
2026-07-21 12:43:27 -07:00
fazerluga-creator
4fda7efcbd fix(mcp): allow background discovery retry after a run that connected nothing
start_background_mcp_discovery() sets _mcp_discovery_started once and never
resets it. If the first background run exits without connecting any MCP
server (startup cancellation, OOM restart, transient network failure), every
later call returns immediately and the process is permanently stuck with
zero MCP tools until a full restart.

Fix: when discovery is marked started but the thread is dead and no server
is connected, reset the flag and spawn a fresh discovery thread. Also log a
WARNING when a discovery run completes with zero connected servers, so the
condition is visible instead of silent.

Caught in production on a long-running gateway fleet where a gateway
restarted under memory pressure and came back with all MCP tools missing.
2026-07-21 12:43:27 -07:00
Teknium
fbe086f7cc fix(mcp): cycle-guard cause/context traversal in transport classifier
Builds on diffen77's #66547 (cherry-picked as the previous commits).
Extend _is_session_expired_error's iterative traversal to follow
__cause__/__context__ in addition to ExceptionGroup .exceptions — SDK
wrappers often raise a generic RuntimeError *from* the message-less
ClosedResourceError, leaving the transport signal reachable only via
the chain. The identity-visited set guards chain cycles (handlers
re-raising previously seen exceptions), and a bounded node budget
(_EXC_TRAVERSAL_MAX_NODES) caps pathological acyclic graphs.

Adds regression tests: cause/context chain detection, interruption
precedence through chains, cyclic cause/context termination, and
budget-bounded termination.
2026-07-21 12:43:27 -07:00
Börje
473545449b fix(mcp): make transport classification cycle safe 2026-07-21 12:43:27 -07:00
Börje
1ae1dd8b2c fix(mcp): harden nested interruption detection 2026-07-21 12:43:27 -07:00
Börje
80209e51db test(mcp): isolate resource error breaker state 2026-07-21 12:43:27 -07:00
Börje
f26fb901ea fix(mcp): reconnect message-less closed transports 2026-07-21 12:43:27 -07:00
Teknium
c4e8ff4d78 chore(contributors): map diffen77 + fazerluga-creator emails
For the #66547 and #66981 salvage cherry-picks in this branch.
2026-07-21 12:43:27 -07:00
Teknium
ee23bffee9 fix(mcp): clear connect-cooldown state on every shutdown path
Builds on trevorgordon981's #50589 (cherry-picked as the previous commit).
The #50394 cooldown reset only ran inside the async _shutdown coroutine,
which is skipped on the empty-_servers fast path — the most common state
when a server failed to connect (failed servers are never recorded in
_servers). It was also skipped when the MCP loop wasn't running.

Clear _server_connect_retry_after/_server_connect_failures on the fast
path and in a final unconditional sweep so a full shutdown/restart always
re-attempts every configured server immediately. Adds regression tests
for both paths.
2026-07-21 12:43:27 -07:00
trevorgordon981
2c3aa3f7b8 fix(mcp): isolate a single failing stdio server from the bridge (#50394) 2026-07-21 12:43:27 -07:00
yungchentang
106d1822e3 fix(mcp): re-register tools during parked revival 2026-07-21 12:43:00 -07:00
Teknium
da8b89c835 fix(mcp): one WARNING per state transition, DEBUG retry chatter, jittered backoff
Log-storm hygiene for the reconnect machinery (#65673, #66092):

- State transitions carry exactly one WARNING each:
  connected → degraded (keepalive failure), degraded → parked
  (budget exhausted / rapid-drop park / permanent error),
  parked → connected (revival proven healthy, via
  _mark_session_proven).
- Per-attempt retry logs (initial-connect attempts, connection-lost
  reconnect attempts, per-cycle rebuild notices, park-wake probes)
  demoted to DEBUG.
- Backoff sleeps get +/-20% uniform jitter (_jittered) so a herd of
  servers that lost the same backend doesn't retry — and log — in
  lockstep.
2026-07-21 12:42:33 -07:00
Teknium
3e6b437762 fix(mcp): unwrap exception groups and park permanent failures immediately (#65673)
- Add module-level _unwrap_exception_group (ported from
  hermes_cli/mcp_config.py, adapted): handles nested groups, prefers
  non-cancellation leaves over the CancelledErrors anyio sprays across
  sibling tasks, and re-raises KeyboardInterrupt/SystemExit leaves.

- Add _classify_mcp_failure(exc) -> 'permanent'|'transient'.
  Permanent: auth 401/403, NonMcpEndpointError, InvalidMcpUrlError,
  FileNotFoundError/ENOENT on the stdio command. Transient: everything
  else (network/EOF/ClosedResource/TaskGroup drops). Permanent failures
  park immediately without burning the retry ladder — every retry
  against a missing binary or a revoked credential hits the same wall.

- Every log site in run() and the keepalive failure log now logs the
  unwrapped root cause as 'TypeName: message', so a dead stdio pipe
  says 'BrokenPipeError' instead of the opaque
  'unhandled errors in a TaskGroup (1 sub-exception)' (and empty
  str(exc) dead-pipe errors are no longer blank).
2026-07-21 12:42:33 -07:00
Teknium
48be1068bb fix(mcp): charge immediate reconnects against a rapid-drop budget (#62212)
Harden the immediate-reconnect path from #66271:

- _reconnect_or_reraise_group re-raises when the transport TaskGroup
  carries a KeyboardInterrupt / SystemExit leaf — fatal signals must
  propagate to the interpreter, never be converted into a reconnect.

- Rapid-drop budget: a completed handshake alone no longer clears
  _reconnect_retries/backoff on clean transport return. A session is
  UNPROVEN until it demonstrates real health — survived >=1 full
  keepalive interval (keepalive success path) or served >=1 successful
  tool call (_mark_session_proven). Unproven clean returns are charged
  against _MAX_RECONNECT_RETRIES, so a flapping transport that
  handshakes fine and drops moments later still reaches the park
  instead of respawning forever (#62212: 6212 spawns in 63h).
  Proven sessions keep the #57604 behaviour: budget clears, transient
  blips over a long-lived session never accumulate toward parking.
2026-07-21 12:42:33 -07:00
hanyu1212
a8a93b6c68 fix(mcp): reconnect immediately on transport TaskGroup drop instead of backoff/park
Streamable-HTTP / SSE MCP transports run their stream pump inside an anyio
TaskGroup. A transient stream drop (idle timeout, brief backend blip,
server-side TCP close) surfaces as a BaseExceptionGroup escaping the
transport context manager. It reached run()'s error path, which applied
exponential backoff (1s..16s) and eventually parked the server for 300s and
deregistered its tools — turning a sub-second glitch into a multi-minute
tool outage even though the POST path was still healthy.

_run_http now wraps all three transport branches (SSE, new + deprecated
Streamable-HTTP) and routes a BaseExceptionGroup through a new
_reconnect_or_reraise_group() helper that returns 'reconnect' (immediate
rebuild, no backoff/park/deregister). It re-raises instead of masking when
shutdown is in progress, when the group carries a real CancelledError
(cancellation must propagate, cf. #9930), or when no live session was
established this attempt (a connect/handshake failure that should fall
through to run()'s backoff rather than hot-loop).

Fixes #66092

Co-authored-by: 雨哥 <hanyu1212@users.noreply.github.com>
2026-07-21 12:42:33 -07:00
Teknium
fb5b8e4c18 chore(contributors): map eazye19@users.noreply.github.com -> eazye19 (PR #42387 salvage) 2026-07-21 12:42:07 -07:00
Teknium
7ae4da28b5 style(cli): open the build lock file with explicit encoding (ruff PLW1514) 2026-07-21 12:42:07 -07:00
Teknium
9adb5a368f test(cli): cover the web UI build flock contention paths
PR #63455 shipped the flock without tests. Cover the three contention
paths: contended+dist serves stale without a second build, uncontended
builds under the lock (creating the lock file), and contended+no-dist
blocks then skips the rebuild because the staleness re-check runs under
the lock and sees the winner's stamp. Also pin the lock filename into
.gitignore via a regression assertion.
2026-07-21 12:42:07 -07:00
Teknium
cca2276115 chore(git): ignore the web UI build lock file
The cross-process build flock (.web_ui_build.lock at the repo root) is
an empty coordination file created on first dashboard boot; it must
never be tracked or show up as untracked noise. Also keeps it out of
the content-hash staleness digest, which skips gitignored paths.
2026-07-21 12:42:07 -07:00
frohsinnllc
37396fc66d refactor(cli): run the web-UI staleness walk once, under the build lock
The flock wrapper checked _web_ui_build_needed twice (pre-lock fast path
and post-lock re-check) and _do_build_web_ui checks it again internally,
so a boot that actually built walked the whole web/ source tree three
times. The callee's own check already runs under the lock on every path
through the wrapper, so it alone is sufficient: drop both wrapper checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 12:42:07 -07:00
frohsinnllc
3a9b2f2f84 fix(cli): serialize concurrent web-UI builds with an exclusive flock
Concurrent dashboard boots (desktop retry loop) each spawned their own
npm install + vite build over the same tree; parallel builds starved
each other, the dist sentinel never advanced, and every boot re-triggered
the build — cascading into orphan backends, port collisions, and CPU
storms that also knocked out the Telegram gateway's heartbeat (2026-07-12).

One process now builds under flock; the rest serve the existing dist
(stale is acceptable) or wait for the first-ever build to finish.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 12:42:07 -07:00
eazye19
e404d6cef9 fix(dashboard): content-hash web UI freshness check
Replace mtime-based web UI dist staleness with a content-hash stamp under HERMES_HOME, matching the desktop build freshness model.

This avoids false stale/fresh decisions when git operations rewrite source mtimes without changing bytes, while preserving the existing stale-dist fallback. Also treats malformed or missing stamp data as needing a rebuild.
2026-07-21 12:42:07 -07:00
Teknium
ad235d95a7 fix(web): guard _serve_index against a missing or unreadable index.html
mount_spa degrades to a JSON 404 catch-all when the dist directory is
fully missing, but _serve_index read index.html unguarded — so a dist
dir that exists while index.html is missing (partial build, wiped dist,
permissions) raised FileNotFoundError on EVERY request instead of
returning a useful error.

Catch OSError around the read and return the same JSON 404 payload the
fully-missing-dist path uses, so clients get a consistent signal. The
route recovers automatically once a rebuild restores the file.
2026-07-21 12:41:41 -07:00
Teknium
18a3fa57bd fix(dashboard): attempt one recovery build when --skip-build finds no dist
--skip-build with a missing web_dist/index.html previously hard-failed
with sys.exit(1) (issue #59288). The desktop launcher passes
--build-mode skip on every boot, so a wiped or never-populated dist
bricked the dashboard until the user manually rebuilt.

Now the default-dist path logs a clear warning and attempts exactly ONE
recovery build through the existing _build_web_ui path. If the recovery
build also fails to produce index.html, the original fatal behavior is
preserved with a clear message. A custom HERMES_WEB_DIST stays fail-fast:
the build writes to the default dist location and cannot populate a
caller-managed directory.

Closes #59288
2026-07-21 12:41:41 -07:00
Aniruddha Adak
26fb0c5d96 fix(packaging): graft web_dist in MANIFEST.in and add sdist regression test
Wheels ship hermes_cli/web_dist via pyproject package-data, but the sdist
did not: MANIFEST.in had no graft and .gitignore excludes web_dist, so
source tarballs installed a dashboard-less package. Graft the directory
and add an sdist regression test that builds the tarball and asserts
index.html is inside.

Salvaged from #29661; the PR's [web]-extra 404-message change was dropped
per maintainer review (misleading guidance for source installs).
2026-07-21 12:41:41 -07:00
Teknium
60cfa11136 feat(kanban): add hermes kanban repair CLI verb
Adds kanban_db.repair_db() — a structured, non-raising wrapper around
the same narrow repair policy as the connect-time guard: probe with
PRAGMA integrity_check under the board's cross-process init flock;
quarantine the corrupt bytes FIRST via the content-addressed backup;
REINDEX only when every integrity message is index-scoped; re-check;
report ok / repaired / corrupt / missing. Locked/busy OperationalError
still propagates raw (a locked healthy DB is not corruption and gets
no quarantine), and a repair invalidates the per-process healthy-path
cache so the next connect() re-probes.

The CLI verb reports status human-readably (or --json), exits 0 for
ok/repaired/missing and 1 when the DB is still corrupt (non-index
corruption stays fail-closed with manual-recovery guidance). It
dispatches BEFORE kanban_command's auto-init: init_db() raises
KanbanDbCorruptError on a corrupt board, which previously would have
made a repair verb unreachable on exactly the boards that need it.

CLI tests drive the real argparse surface (build_parser +
kanban_command) against real corrupted SQLite fixtures.
2026-07-21 12:41:14 -07:00
Teknium
49828a3fd6 feat(kanban): periodic WAL checkpoint (TRUNCATE) on the dispatcher tick
Kanban connections set wal_autocheckpoint=100, but SQLite's passive
autocheckpoint backs off whenever any reader holds an open snapshot —
on a busy multi-process board the -wal file can grow without bound
between gateway restarts.

After each successful dispatch tick, while still holding the board's
single-writer dispatch flock, run PRAGMA wal_checkpoint(TRUNCATE)
best-effort at a coarse interval (>=5 min since this process last
checkpointed that board; module-level per-path monotonic timestamp, so
multi-board dispatchers checkpoint each board on its own clock).
Success and busy/locked skips are both logged at DEBUG; a failing
checkpoint can never fail the tick.
2026-07-21 12:41:14 -07:00
Teknium
8fb3cc1b1a fix(kanban): cap corrupt-backup retention at 10 files per board DB
Content-addressed quarantine backups dedupe identical corrupt bytes,
but corruption that keeps mutating between failures (partial repairs,
further damage across dispatcher retries, multi-profile fleets) mints a
new sha-named backup every round — a user accumulated 124
.corrupt.*.bak files with no bound.

After each NEW backup is created, prune oldest-by-mtime backups beyond
_CORRUPT_BACKUP_RETENTION (module constant, default 10), including the
copied -wal/-shm sidecars. The just-created backup is always exempt
(copy2 preserves the source mtime, which can be older than existing
backups). Pruning is best-effort and never masks the corruption error
about to be raised; dedupe of identical corrupt bytes is unchanged.
2026-07-21 12:41:14 -07:00
Teknium
8995131458 fix(kanban): auto-repair index-only kanban.db corruption via REINDEX
_guard_existing_db_is_healthy previously failed closed on ANY
integrity_check failure, including the index-scoped class ('wrong # of
entries in index <name>' / 'row N missing from index <name>') where the
table b-trees are intact and REINDEX rebuilds the damaged indexes
losslessly. Boards hit by that class were bricked until manual surgery
even though SQLite can fix them in-place.

Now, when integrity_check output consists ONLY of index-scoped errors
(index name parsed generically from the message — no hardcoded list):

  1. quarantine the corrupt bytes FIRST via the existing content-
     addressed _backup_corrupt_db,
  2. under the caller-held cross-process init flock, REINDEX each named
     index (falling back to bare REINDEX if a parsed name doesn't
     resolve),
  3. re-run integrity_check and proceed only if it comes back clean.

Any non-index error class (page corruption, malformed image, freelist
damage) — or a REINDEX whose re-check is still dirty — fails closed
exactly as before: backup + KanbanDbCorruptError, no silent recreation.
Transient OperationalError (locked/busy) still propagates raw with no
quarantine.

Tests build a real board DB and corrupt a live index via the
writable_schema/partial-index REINDEX trick to produce the genuine
'wrong # of entries in index' shape, then assert auto-repair recovers
with data intact, page corruption still raises, and a dirty re-check
fails closed.
2026-07-21 12:41:14 -07:00
Teknium
2d6b95cf2f test(state): exercise REINDEX repair against a REAL stale B-tree index
Replace the mocked test for #63398's REINDEX strategy: the original
monkeypatched _db_opens_cleanly to return the corruption string, so the
REINDEX pass itself was never exercised against actual index corruption —
the test would pass even if REINDEX didn't fix anything.

New fixture _corrupt_btree_index() builds genuine on-disk staleness with a
writable_schema hack: rewrite the index definition to a partial index
(WHERE 0), REINDEX so the b-tree is rebuilt empty, then restore the full
definition. integrity_check then reports the real
'wrong # of entries in index idx_messages_session' / 'row N missing from
index' class from #63386 — no mocks anywhere.

The rewritten test asserts end-to-end with real function calls:
- the real _db_opens_cleanly detects the stale index,
- repair_state_db_schema repairs it with strategy 'reindex_btree',
- post-repair the detector and raw PRAGMA integrity_check both report
  healthy, and a query forced through the rebuilt index (INDEXED BY) sees
  every row.

Adds a second test asserting the REINDEX strategy is non-destructive
(all sessions/messages survive, readable via SessionDB).

Follow-up to #63398; refs #63386
2026-07-21 12:40:48 -07:00
liuhao1024
505fb58751 fix(state): add REINDEX strategy to repair stale B-tree indexes (#63386)
When PRAGMA integrity_check reports 'wrong # of entries in index' for
B-tree indexes (e.g. idx_sessions_handoff_state), the existing repair
strategies (FTS rebuild, sqlite_master dedup, drop-FTS+VACUUM) don't
address the mismatch. Add Strategy 0.5: run REINDEX to rewrite the
index b-tree from canonical table rows before escalating to more
destructive strategies.
2026-07-21 12:40:48 -07:00
Teknium
373ec23e37 fix(state): extend search-path FTS self-heal to the CJK/trigram branch
The trigram MATCH branch in search_messages() had the same
OperationalError-only catch that #66420 fixed on the main FTS5 branch: a
corrupt messages_fts_trigram shadow table raises the malformed /
'fts5: corrupt structure record' class (sqlite3.DatabaseError, parent of
OperationalError), which propagated straight out of search_messages and
crashed CJK session/history search for read-only sessions.

Route that class through the shared one-shot _try_runtime_fts_rebuild()
and retry the trigram query (catch moved outside self._lock so
rebuild_fts() can re-acquire it, mirroring the main branch). If the
rebuild is refused (guard consumed / FTS disabled / different error) or
the retry fails, fall through to the existing LIKE substring fallback —
which reads only the canonical messages table — instead of raising, so
CJK search degrades gracefully rather than crashing.

Adds two regression tests: trigram search self-heals in place after
shadow-table corruption (answers from the rebuilt trigram index, not the
LIKE fallback), and degrades to LIKE without raising when the one-shot
rebuild was already consumed.

Follow-up to #66420; refs #66296 #66724
2026-07-21 12:40:48 -07:00
Frowtek
11710c51fc fix(state): self-heal FTS corruption on the SessionDB search path too
Complements #66296 (self-heal on the write path): search_messages()'s main
FTS5 MATCH query caught only sqlite3.OperationalError (a query-syntax error →
return empty). A corrupt FTS index raises the malformed / "fts5: corrupt
structure record" class, which is a sqlite3.DatabaseError — the parent of
OperationalError, so it was NOT caught and propagated straight out of
search_messages, crashing session/history search.

The write path now rebuilds and retries on that class, but a read-only
session (cron/CLI history search, or a search issued before any write) never
triggers a write, so its search stayed broken until the next process restart
ran the offline repair.

Catch the DatabaseError corruption class on the search MATCH read too and
route it through the existing one-shot _try_runtime_fts_rebuild(), then retry
the query. The catch is moved outside `with self._lock` so rebuild_fts() can
re-acquire the lock (mirrors _execute_write). The one-shot guard is shared
with the write path, so a single instance never loops on a genuinely
unrecoverable index. OperationalError syntax handling is unchanged (caught
first).

Adds a regression test: with a corrupted messages_fts and no post-corruption
write, search_messages() rebuilds in place and returns the match; without the
fix it raises DatabaseError.
2026-07-21 12:40:48 -07:00
Enough1122
96c1511495 fix(state): preserve degraded-runtime read probe + use canonical FTS5 classifier
Two follow-ups on top of f842733 (the FTS5 read probe added in #66906):

1. The original probe query used MATCH '', which FTS5 rejects with
   'fts5: syntax error near '. Empty MATCH syntax is not valid FTS5.
   Switch to MATCH '""' — a quoted empty phrase that parses, scans
   zero rows, and exercises the same shadow-table read path the
   search tools use. The probe previously never reached the shadow
   segments at all on a healthy DB; the read-corruption class was
   only being detected because the existing write probe happens to
   fail first on a DatabaseError.

2. The probe's degraded-runtime branch only checked the substrings
   'no such table' / 'no such column'. On a SQLite build without the
   fts5 module, MATCH against a legacy messages_fts table raises
   'no such module: fts5' (a different OperationalError class). The
   substring check would misclassify that as corruption and trigger
   repair, whose final fallback deletes the messages_fts% schema
   (#66906 review). Use SessionDB._is_fts5_unavailable_error() — the
   canonical classifier already used by the degraded-runtime init
   path — to recognize both 'no such module: fts5' and
   'no such tokenizer: trigram' as capability errors.

Add tests covering:
- Partial shadow-table damage (read-corruption class)
- Repair brings reads back online
- Healthy degraded DB without fts5 module stays healthy (regression
  for the misclassification risk)
- Healthy degraded DB without trigram tokenizer stays healthy

Closes #66906 review feedback
Refs #66724
2026-07-21 12:40:48 -07:00
Enough1122
57b3c477c0 fix(state): also catch sqlite3.DatabaseError in FTS5 read probe (#66724)
The FTS5 read probe in _db_opens_cleanly() only caught
sqlite3.OperationalError. But the corruption class #66724 actually
wants caught — partial shadow-table damage where MATCH / snippet / rank
queries raise DatabaseError("database disk image is malformed") — is a
DatabaseError, not OperationalError. Without this catch the probe
crashes the caller instead of returning a reason, which is exactly the
silent-fail mode the issue describes.

Move the try/except inside the for-loop so each FTS table is probed
independently (one table corrupted should still surface as a reason),
add a separate except clause for DatabaseError that surfaces the same
reason format, and use continue instead of pass so the loop still walks
both tables when only one is missing on a brand-new DB.

Tested by hand: with a corrupted messages_fts_trigram shadow table the
function now returns 'fts5 read probe failed on messages_fts_trigram:
database disk image is malformed' instead of crashing out. Without this
fix it would still crash.
2026-07-21 12:40:48 -07:00