zxell.ai Tokenizer Comparison Report (regenerated)

Date: 2026-09-27 ((omitted: how the original version was lost) This document was regenerated from the models retrained on the new server on 2026-09-05. The numbers match the report as of 8/13 to the third decimal place.)

Scope: training and comparison of SentencePiece BPE at 48k / 64k with zxell_prep/train_compare_tokenizers.py. Models: sp_bpe_48k.model (adopted) / sp_bpe_64k.model (for comparison). Text listing of the vocabulary: sp_bpe_48k.vocab (48,000 lines, "piece␉score").

1. Conclusion

The 48k vocabulary is adopted. The estimated token count of the whole corpus is 6.14B (measured: 6.152B, fixed by the full tokenization on 9/5).

64k reduces the total token count by only 3% (0.18B) while making the embedding layer 33% larger (in the S model, 48k×384 = 18.4M parameters becomes 64k×384 = 24.6M, so the embedding takes up most of the 30M total). On the premise of running S on a GTX 1050 with 2GB of VRAM, 48k is the reasonable choice. Even now that an RTX 3060 has joined, the picture is the same — a bloated embedding squeezes the "learning capacity of the body" of S/M — so the decision stands.

2. Training data (samples)

Extracted from the snapshot with extract_sample.py (5% of every language by id%20==0, plus ja raised to id%4==0). Measured on 9/5 (within 0.2% of the old 8/13 version taken from the live DB):

567,982 rows extracted / 2,164 duplicate links / 2,202 empty / 315 seconds (new server).

The tokenizer training file sp_train.txt is apportioned by a per-language character cap (en 120M / de 85M / fr 70M / ja 65M characters. 340,379,150 characters in total, 192,625 sentences). At the natural ratio ja would make up only 1.5% and the Japanese vocabulary would not develop, so for tokenizer training only, ja is thickened to about 19% (this is separate from how the training data itself is apportioned).

3. Training settings

model_type=bpe, character_coverage=0.9995, byte_fallback=True (fall back to bytes instead of emitting UNK)
split_digits=True (digits one at a time), normalization_rule_name=nmt_nfkc
max_sentence_length=16384, input_sentence_size=2,000,000, shuffle=True

Training time (Ryzen 3 4300U, with AVX2): about 84 seconds for 48k / about 86 seconds for 64k (about 4 minutes each on the old Celeron N3350). Alphabet size 2,177 / character coverage 99.95%.

4. Compression efficiency (evaluation set, not used for training)

The estimate is: total characters per language measured in phase 0 × the 0.947 survival rate after deduplication ÷ chars/token. The full tokenization on 9/5 measured 6.152B (0.2% off the estimate), with a language mix of en 64.6% / de 21.7% / fr 10.0% / ja 3.7% (by tokens).

5. What the vocabulary (48k) contains

Breakdown of the 48,000 pieces by script (classified mechanically):

Japanese pieces account for about 18,000 in total (37%). This is the effect of oversampling ja.

6. Segmentation examples (48k model)

en: The central bank raised interest rates for the third time this year.
    ␣The | ␣central | ␣bank | ␣raised | ␣interest | ␣rates | ␣for | ␣the | ␣third | ␣time | ␣this | ␣year | .
    68 characters → 13 tokens (5.23 chars/token)

de: Die Bundesregierung hat ein neues Konjunkturpaket beschlossen.
    ␣Die | ␣Bundesregierung | ␣hat | ␣ein | ␣neues | ␣Konjunktur | p | aket | ␣beschlossen | .
    62 characters → 10 tokens (6.20). The compound Konjunkturpaket is split in 3

fr: Le gouvernement a annoncé de nouvelles mesures économiques mercredi.
    ␣Le | ␣gouvernement | ␣a | ␣annoncé | ␣de | ␣nouvelles | ␣mesures | ␣économiques | ␣mercredi | .
    68 characters → 10 tokens (6.80)

ja: 政府は水曜日に新しい経済対策を発表した。
    ␣政府は | 水曜 | 日に | 新しい | 経済 | 対策を | 発表した | 。
    20 characters → 8 tokens (2.50). Cut at meaningful units: 「政府は」, 「対策を」, 「発表した」

Short, tidy sentences come out more efficient than the evaluation-set averages (4.2 / 4.2 / 3.8 / 1.9). The evaluation set contains URLs, numbers, proper nouns, and HTML residue, so those averages are the effective values.

7. Known caveats

• ja's 1.95 chars/token is less than half that of the other languages. The 0.4G characters of ja come to only 0.24B tokens, and the share of ja in the training data is 3.7% by tokens. Japanese capability has to be evaluated on the premise that "the data is thin"
• Duplicated concatenation in ja articles (review19/20): with a naive concatenation of title + description + content, many articles repeated the same sentence 2–3 times. On 2026-09-27 textprep.compose_ja was improved so that "a part already contained at the start of what follows is not concatenated" (the measured effect is in review20). After re-encoding, the ja token count is expected to drop by a few percent to a dozen or so percent
• HTML residue: in the bodies of some feeds, tags such as <p><img src=…> remain as raw text. The size survey and the fix are a question in review20
• The evaluation sets for de/fr are somewhat small at 2,028 / 1,094 articles (a 5% sample × the cap of 1 in 50 articles). That is enough for estimating chars/token, but for evaluating the model (perplexity) the val/test shards are used instead