mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-07-23 16:36:23 +00:00
The salvaged estimator ran a per-character Python loop on every
estimate_tokens_rough() call — a ~28,000,000x slowdown vs (len+3)//4 on a
1MB ASCII tool output (measured ~3.0s per call). Gate it:
- str.isascii() O(1) fast path keeps pure-ASCII text bit-identical to the
classic (len+3)//4 rule at ~1.3x baseline cost (0.23us vs 0.17us per
1MB call).
- Non-ASCII text counts dense CJK chars via a compiled character-class
regex in C (len(text) - len(re.sub(''))): ~352ms/1MB hangul vs ~2.1s
for the per-char loop.
- Non-ASCII-but-non-CJK text (accents, Cyrillic, emoji) keeps the classic
rule.
Also: parity tests against the per-char reference implementation, and
updated two stale expectations that encoded the old behavior (CJK now
counted ~1 token/char; short string content now ceil-divided instead of
floored to 0). The continuity test now detects merged-into-tail summaries
via _is_context_summary_content.
|
||
|---|---|---|
| .. | ||
| emails | ||
| README.md | ||
Contributor email → GitHub login mappings
This directory replaces appending entries to AUTHOR_MAP in
scripts/release.py. The old dict caused constant merge conflicts when
several salvage PRs landed at once — every PR edited the same lines of the
same file. Here, each mapping is its own file, and file additions never
conflict.
Adding a mapping
One file per commit-author email, under emails/:
python3 scripts/add_contributor.py <email> <github-login>
# or by hand:
echo "<github-login>" > contributors/emails/<email>
- File name = the exact commit-author email (as shown by
git log --format='%ae'). - File content = the GitHub login on the first non-comment line.
Lines starting with
#are comments (use them for the PR reference).
Example — contributors/emails/jane.doe@example.com:
janedoe
# PR #12345 salvage (gateway: fix session key routing)
Rules
- Do NOT add new entries to
AUTHOR_MAPinscripts/release.py. That dict is frozen legacy data; the release tooling merges it with this directory (directory entries win on duplicates). - GitHub noreply emails (
<id>+<login>@users.noreply.github.comand<login>@users.noreply.github.com) auto-resolve — no file needed. - The
Contributor Attribution CheckCI job fails a PR whose commits carry an unmapped email; the failure message prints the exact command to run.