The char-level streaming fuzz runs 12 seeds × 500 growing prefixes
(~6000 full+cached lexes) and first trips the pre-fix boundary at
seed 11 / step 257, so the workload can't shrink without gutting the
guard. The work is bounded but exceeds Vitest's 5s per-test default on
CI workers under parallelism, so give this one test an explicit 30s
timeout instead of weakening coverage.
The streaming block splitter added in #67154 dropped only the previous
parse's trailing whitespace blocks plus its LAST content block before
re-lexing the appended suffix. That boundary is unsound: a trailing
Setext underline (`-`/`=`) underlines the paragraph ABOVE it, so
appending to it can retroactively merge the previous parse's last TWO
blocks into one.
Minimal repro: cached "…#e\n5\n-" lexes to [ …, "#e\n", "5\n-" ], but
grown to "…#e\n5\n-p2=kj:c" collapses "#e"/"5\n-" into a single block.
The reused settled prefix still contained a stale "#e\n" block. The
`blocks.join('') === text` guard can't detect this because the wrong
split reconstructs the same source string, so the divergence rendered
as mis-split blocks with no fallback.
Fix: drop the last TWO content blocks (skipping whitespace-only blocks
around them) before re-lexing the suffix. The block before the last is
the deepest an append can reach — a Setext underline consumes exactly
one preceding block — and earlier blocks stay fenced off by settled
blank lines, so re-lexing two is sufficient and safe.
Tests: a deterministic regression for the exact prev→grown pair, and a
character-level streaming property fuzz (12 seeds × 500 growing prefixes
over the markdown control alphabet). Both fail on the pre-fix boundary
and pass after. tsc/eslint/prettier clean; markdown-text suite green.
Fourth profiling round (#66033/#66347/#66470). Streamdown's
parseMarkdownIntoBlocks is a full `marked` lex of the entire message,
and during streaming every flush is a new string — so the splitter paid
O(full-text) ~30x/s: benchmarked 3.4-9.6ms per call at 64-192KB, i.e.
15-30% of a core burned re-lexing settled text on long agent replies.
(The rest of the May-2026 "re-parse elephant" was already eaten by
tailBoundedRemend, block-memo, the KaTeX memo, and deferred shiki —
the splitter was the last O(full-text) pass besides preprocess, which
benches at only 1.5-4.6ms and is left alone.)
New src/lib/markdown-blocks.ts (extracted from markdown-text.tsx) wraps
the splitter with two caches:
- the existing exact-string LRU (remounts: virtualizer scroll, session
switch) — moved, unchanged;
- a streaming-append cache: when the new text startsWith a recently
parsed text, reuse that parse's blocks up to a settled boundary and
lex only the suffix. The boundary drops trailing whitespace-only
blocks plus the last content block — the only block appended text can
reinterpret (open fence, list/table continuation, setext underline,
lazy blockquote). Earlier blocks are separated by settled blank lines
and can't change. Cross-block reference links can't regress:
Streamdown already renders each block as an independent document.
Safety: the splitter's blocks.join('') === text property makes offsets
exact and is defensively re-checked; any mismatch or non-append rewrite
(edit, branch swap) falls back to the full lex — byte-for-byte the old
behavior.
Property tests: at every random streaming cut over a corpus covering
fences, loose lists, tables, setext headings, lazy blockquotes, HTML
blocks, and display math — and token-by-token through a fence boundary
— the cached splitter's output is asserted deep-equal to a fresh full
lex. Bench (128KB reply, 200 flushes): 990ms -> 72ms total splitter CPU
(4.95 -> 0.36ms/flush).
Verification: tsc clean; eslint/prettier clean; lib + assistant-ui
suites green (473 tests).
