mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-28 18:19:28 +00:00
f13f845116
Integration layer for the cjk_unicode61 tokenizer, rebuilt on the v23 schema (the contributed integration in PR #65544 predated it): - messages_fts_cjk: external-content FTS5 over a tool-row-excluding view (same v23 storage discipline as the trigram index it supersedes — zero inline text copies). Serves EVERY CJK query shape the legacy routing split between trigram (>=3 chars/token) and LIKE full scans (1-2 char tokens). Lone 1-char CJK runs and role_filter=['tool'] queries keep their legacy routes. - Dedicated marker pair (fts_cjk_rebuild_high_water/progress) gates the id-scoped triggers, so a cjk-only backfill never gates the complete messages_fts/trigram triggers. - Transitions ride (the existing throttled/resumable chunk engine): fresh DBs are born with the index; legacy v22 DBs land on v23+cjk in one run; already-optimized v23 DBs gaining the tokenizer get a marker-gated backfill; live writes are indexed immediately in every case. - Tokenizer-loss self-heal: a process that can't load the extension drops the cjk triggers (writes keep working), leaves a stale breadcrumb, and the index is rebuilt from scratch on the next optimize run — triggers are never reinstalled over a gap (external-content 'delete' on an unindexed rowid is the FTS5 corruption hazard the marker gating exists to prevent). - Capability classification: 'no such tokenizer: cjk_unicode61' joins the degraded-runtime error class everywhere (read probe, write probe, repair) so tokenizer absence is never misclassified as corruption. - Config: sessions.cjk_fts (default on, inert without the .so) and sessions.search_slow_ms in config.yaml, bridged to env by CLI + gateway (startup + per-turn reload). build.sh falls back to vendored SQLite headers so no libsqlite3-dev is needed. Slow-query log path attribution updated: fts_cjk / fts5 / trigram / like_scan. Tests: 14 lifecycle tests (fresh/legacy/stale/backfill paths, tokenizer-loss round-trip) + 5 config-bridge tests + slow-log suite.
24 lines
860 B
Markdown
24 lines
860 B
Markdown
# fts5_cjk — cjk_unicode61 FTS5 tokenizer
|
|
|
|
unicode61 + CJK character bigrams (Lucene CJKAnalyzer semantics). Fixes
|
|
1-2 char Korean/Chinese/Japanese terms falling through to LIKE full-table
|
|
scans in session search.
|
|
|
|
Build & install to `~/.hermes/lib/`:
|
|
|
|
./build.sh
|
|
|
|
Uses the system `sqlite3ext.h` when available, else the vendored copy in
|
|
`vendor/` — no libsqlite3-dev required.
|
|
|
|
Once the extension is installed, the next `SessionDB` open creates the
|
|
`messages_fts_cjk` index (external-content, tool rows excluded — same v23
|
|
storage discipline as the other indexes). On a populated database, run
|
|
|
|
hermes sessions optimize-storage
|
|
|
|
to backfill it; new messages are indexed live either way. Set
|
|
`sessions.cjk_fts: false` in `~/.hermes/config.yaml` to disable. Override
|
|
the .so location with `HERMES_FTS5_CJK_SO`.
|
|
|
|
Contributed by Soju06 (PR #65544).
|