Soju06 b10952e9c6 feat(state): cjk_unicode61 FTS5 tokenizer — unicode61 + CJK bigrams (native extension) ... unicode61 indexes a CJK run as ONE token, so 2-char Korean terms (일본, 구글, 우리, ...) can never match it and the trigram tokenizer needs >=3 chars per term — any query containing a 1-2 char CJK token falls through to a LIKE full-table scan (measured 3-6.4s CPU per query on a 6.8GB production state.db; the #1 base cost behind a 12.4s session_search average on CJK workloads). This ships a ~250-line loadable FTS5 tokenizer (no deps) that wraps unicode61: maximal CJK runs inside its tokens are re-emitted as overlapping character bigrams (Lucene CJKAnalyzer semantics), everything else passes through unchanged. FTS5 phrase semantics turn consecutive sub-tokens into exact substring matching down to 2-char terms at index speed. Build: native/fts5_cjk/build.sh -> ~/.hermes/lib/libfts5_cjk.so (override: HERMES_FTS5_CJK_SO). Salvaged from PR #65544; the schema integration lands separately on the v23 external-content layout.		2026-07-22 07:56:47 -07:00
..
fts5_cjk	feat(state): cjk_unicode61 FTS5 tokenizer — unicode61 + CJK bigrams (native extension)	2026-07-22 07:56:47 -07:00