fix(cli,tui): align CJK / wide-char markdown tables (#23863)

CJK and emoji glyphs render as two terminal cells but JS String#length
and the model's own padding count them as one, so any markdown table
with Chinese / Japanese / Korean cells drifts right per row when a
real terminal renders it. Both surfaces fix this with a display-cell
width measurement (wcswidth on the Python side, stringWidth on the
TUI side).

Changes:
- agent/markdown_tables.py: new helper. realign_markdown_tables(text)
  detects markdown table blocks (header + |---| divider) and
  rewrites the row padding using wcwidth.wcswidth so every pipe and
  dash lines up across rows. No-op on text without tables.
- cli.py: hook the helper into _render_final_assistant_content for
  strip / render modes (raw passes through untouched), and into the
  streaming line emitter so live token-by-token rendering also
  produces aligned tables. A small two-buffer state machine in
  _emit_stream_text holds table rows until the block ends, then
  flushes them through the realigner so all rows pad to a single
  per-column width.
- ui-tui/src/components/markdown.tsx: renderTable now uses
  stringWidth (Bun.stringWidth fast path + East-Asian-width-aware
  fallback, already memoised in @hermes/ink) instead of UTF-16
  String#length for both column-width measurement and per-cell
  padding. Drops the comment that documented the bug as a deliberate
  limitation.

Validation:
- New tests/agent/test_markdown_tables.py (11): every rebuilt block
  shares pipe column offsets across rows for pure CJK, mixed
  CJK+emoji, ragged-row, and multi-table inputs.
- Updated tests/cli/test_cli_markdown_rendering.py: the existing
  strip-mode test asserted exact whitespace; rewritten to assert the
  alignment contract (cell content survives + every rendered row
  shares pipe offsets).
- New ui-tui markdown.test.ts case (1): rendered column-2 start
  offset is identical for the header + every body row, including
  the CJK row that drifted before the fix.
- Live: hermes chat -q with the user-reported screenshot prompt now
  produces a perfectly aligned table on the wire (header, divider,
  4 body rows including '通义千问', all pipes at identical columns).
This commit is contained in:
Teknium 2026-05-11 11:13:06 -07:00 committed by GitHub
parent 657874460f
commit 1d00716754
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 559 additions and 15 deletions

View file

@ -118,14 +118,37 @@ def test_strip_mode_preserves_table_structure_while_cleaning_cell_markdown():
)
output = _render_to_text(renderable)
assert "| Syntax | Example |" in output
assert "|---|---|" in output
assert "| Bold | bold |" in output
assert "| Strike | strike |" in output
# Inline cell markdown is stripped (the contract this test enforces).
assert "**" not in output
assert "~~" not in output
assert "`" not in output
# Cell *content* survives, even if the surrounding whitespace was
# rewritten by the wcwidth-aware re-aligner. Asserting on bare
# cell text keeps this test focused on the strip behaviour rather
# than snapshotting incidental column padding (which is what the
# CJK-alignment fix changes).
assert "Syntax" in output
assert "Example" in output
assert "Bold" in output and "bold" in output
assert "Strike" in output and "strike" in output
# Structural sanity: the table still renders as pipe-bordered rows
# (header + divider + 2 body rows).
body_rows = [ln for ln in output.splitlines() if ln.strip().startswith("|")]
assert len(body_rows) == 4
# Every rendered table row shares the same pipe column offsets — the
# alignment guarantee from realign_markdown_tables.
pipe_cols = [
[i for i, ch in enumerate(row) if ch == "|"] for row in body_rows
]
assert all(p == pipe_cols[0] for p in pipe_cols), (
"table rows misaligned after strip-mode rendering:\n"
+ "\n".join(body_rows)
)
def test_final_assistant_content_can_leave_markdown_raw():
renderable = _render_final_assistant_content("***Bold italic***", mode="raw")