A tokenizer is the boundary between human text and a language model’s vocabulary. It normalizes and splits raw Unicode text into tokens, then maps those tokens to integer IDs. The transformer itself does not reason over characters directly: those IDs are looked up in an embedding table, processed as vectors, and later decoded back through the same vocabulary.
Discrete building blocks. Like atoms, tokens are the discrete units the model can address by number. Every input and output is a flat sequence of integer IDs, one per token.
Irreducible basis. Every integer is built by multiplying primes together. The model's job is to learn the "grammar of token combinations" just as number theory studies prime combinations.
The tokenizer splits raw text into pieces, then maps each piece to an integer. The model never sees letters — only those numbers.
Context windows and API costs are counted in tokens. A Nepali-aware tokenizer packs the same text into far fewer tokens than a generic one.
A tokenizer trained on Nepali text may keep नमस्ते as one unit. A generic tokenizer with no Nepali in its training corpus may split the same word into five byte-sized fragments — five IDs, five embedding lookups, five attention positions.
Pick a sample and press Tokenize! Each highlighted chunk is one token — one vocabulary entry the model can address by ID. This demo uses hand-drawn splits to teach the idea. For real tokenizer output on Nepali corpus sentences, use the Token visualization just below.
Tokenization happens before the transformer reads text and after the transformer predicts output IDs. The tokenizer is not the model’s reasoning engine, but it sets the alphabet of possible moves. Change the tokenizer after training and the learned embedding rows no longer mean what the model expects.
Sample
Raw text
Hello, world!
Tokens (pieces)
Ġworld — where did the space go?
GPT-family tokenizers (BPE-style) do not emit a standalone space token followed by world. Instead, the space before a word is merged into the first piece of that word during training. Tools display this merged-in space as the Unicode character Ġ (U+0120). So Ġworld is one token, representing space + world — the space has not vanished, it is encoded inside.
The blue SPACE character is absorbed into the highlighted Ġworld token. No information is lost — Ġ is just the display convention that says "this token starts a new word after whitespace."
If you write Hello, world ! with extra spaces or different punctuation, the tokenizer may split differently — additional space tokens may appear, or punctuation may shift boundaries. That is expected: BPE merge rules were learned from specific whitespace patterns in the training corpus.
Interactive: choose a corpus sentence, then turn tokenizers on or off to compare side by side. These are real splits from seven benchmark tokenizers — the same sentences listed in the corpus section further down.
Look for boundary conventions: ## marks WordPiece continuations, ▁ marks a SentencePiece word boundary, and special tokens such as [CLS], [SEP], <cls>, or <sep> are real sequence pieces for those setups. Byte-level BPE pieces may look like character fragments because Devanagari code points are represented through byte-derived merges.
Recommended path: try the atom lab, then play with real tokenizer splits. When you want to understand the numbers, read metrics → models → corpus → results → charts → honorifics → score sheet.
Use these cards as a guided route. The sticky bar appears while you scroll and keeps the same order.
These metrics describe the tokenizer output, not the intelligence of the model using it. They are only comparable when the same text, normalization, word counting, and special-token policy are used for every tokenizer.
Fertility
Average number of tokens produced for each counted word.
Think of fertility as "boxes per item". You want to pack your words into as few token boxes as possible.
Caveat: 1.0 is not automatically best. Good subword suffixes can help; special tokens like [CLS]/[SEP] can also raise counts.
Chars / token
Average number of Unicode characters packed into each token.
Think of tokens as vehicles. A bus (high chars/token) carries many characters efficiently. Single-occupant cars (low chars/token) cause traffic jams in the model.
Caveat: very large tokens are not always better. A coarse token can hide suffixes, morphology, or spacing distinctions the model may need.
NSL (normalized sequence length)
Sequence length compared to a baseline tokenizer for the same sentence.
Think of NSL as packing efficiency compared to a baseline. 0.8x is 20% more efficient, while 2.5x takes up 2.5 times more space.
Caveat: NSL is relative. Changing the baseline changes the number, so use it to compare tokenizers within the same experiment.
Speed
How many tokens per second the tokenizer implementation processed in this benchmark run.
Think of speed as a factory conveyor belt. A slow tokenizer is a bottleneck, delaying the powerful AI model waiting for the text to be converted to numbers.
Caveat: speed depends on implementation details: hardware, batching, caching, Python vs Rust backend, and vocabulary size.
GPT family encodings are byte-level BPE (tiktoken). They were optimised for English-heavy web text: Devanagari Nepali often splits into many small pieces compared with Indic-pretrained models.
import tiktoken
enc = tiktoken.get_encoding("gpt2")
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
gpt2 and better multilingual coverage in practice, but it is still not a Nepali-specific vocabulary. GPT-4o uses a newer encoding, so this benchmark should be read as cl100k_base, not GPT-4o itself.gpt2 on non-Latin text, but still not designed around Devanagari morphology.
Indic checkpoints differ in whether Nepali was in the pretraining mix. MuRIL explicitly lists Nepali (ne) among its supported languages. The public IndicBERT and IndicBART cards do not expose Nepali as a primary supported language/tag in the same way, but their tokenizers still encode Devanagari strings and are useful Indic subword comparisons.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ai4bharat/indic-bert")
AutoTokenizer; the visible ▁ marks word starts. Check the model revision and tokenizer files before comparing runs.from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("google/muril-base-cased")
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
"ai4bharat/IndicBART",
do_lower_case=False,
use_fast=False,
keep_accents=True,
)
AlbertTokenizer in older examples). Language tags like <2hi> matter for training-time format.# Your pipeline, e.g. tok = WordPiece()
# Your pipeline, e.g. tok = SentencePiece()
.model), vocabulary size, and whether normalization NFC/NFD is applied before encode.Subtle point: a Nepali-specific tokenizer is not automatically better for every model. It is better only if its vocabulary, normalization, and special-token policy match the model you train or adapt. Once a model has learned embeddings for one vocabulary, you cannot swap in a different tokenizer without also changing and retraining the affected embedding/output layers.
Fixed strings for comparing honorific register, plain sentences, news-like complexity, technical vocabulary, and literary style. The point is controlled comparison: every tokenizer sees the same Unicode strings, including punctuation and special marks such as the zero-width non-joiner in संसद्मा.
Group 1 — Honorifics (same meaning, four respect levels)
तँ भात खान्छस् ।
तिमी भात खान्छौ ।
तपाईं भात खानुहुन्छ ।
हजुर भात खानुहुन्छ ।
तँ कहाँ जान्छस् ?
तिमी कहाँ जान्छौ ?
तपाईं कहाँ जानुहुन्छ ?
हजुर कहाँ जानुहुन्छ ?
Group 2 — General Nepali
नमस्ते, तपाईंलाई कस्तो छ ?
रामले भात खायो ।
आज मौसम राम्रो छ ।
Group 3 — News / complex
नेपाल सरकारले नयाँ शिक्षा नीति लागू गर्ने निर्णय गरेको छ ।
काठमाडौं उपत्यकामा यातायात व्यवस्थापनका लागि नयाँ योजना तयार पारिएको छ ।
प्रधानमन्त्रीले संसद्मा नयाँ बजेट प्रस्तुत गरे ।
Group 4 — Technical
कृत्रिम बुद्धिमत्ता र मेसिन लर्निङले नेपाली भाषा प्रशोधनमा नयाँ आयाम थपेको छ ।
डिजिटल प्रविधिको विकाससँगै सूचना प्रविधि क्षेत्रमा रोजगारीका अवसर बढेका छन् ।
Group 5 — Literary
हिमालयको काखमा बसेको यो सुन्दर देशले विश्वलाई आफ्नो प्राकृतिक सौन्दर्यले मोहित पारेको छ ।
मानिसको जीवनमा सफलता पाउन परिश्रम, धैर्य र दृढ संकल्पको आवश्यकता हुन्छ ।
Aggregates from encoding the full evaluation corpus above. Rank orders by total tokens: fewer tokens means shorter model sequences for this corpus. That is useful, but not the whole story. A tokenizer also needs low unknown-token rates, stable normalization, and pieces that preserve distinctions important for the downstream model.
| Rank | Tokenizer | Total tokens | Fertility ↓ | Chars/token ↑ | Avg tok/sent | Max tok/sent | Speed (tok/s) ↑ |
|---|---|---|---|---|---|---|---|
| 1 | MuRIL | 154 | 1.23 | 3.9 | 8.6 | 18 | 78796 |
| 2 | Nepali WP | 183 | 1.46 | 3.28 | 10.2 | 21 | 124887 |
| 3 | Nepali SP | 192 | 1.54 | 3.12 | 10.7 | 23 | 180428 |
| 4 | IndicBART | 249 | 1.99 | 2.41 | 13.8 | 31 | 71172 |
| 5 | IndicBERT | 253 | 2.02 | 2.37 | 14.1 | 29 | 107245 |
| 6 | GPT-4 / GPT-3.5 (cl100k) | 727 | 5.82 | 0.83 | 40.4 | 87 | 466440 |
| 7 | GPT-2 | 1136 | 9.09 | 0.53 | 63.1 | 147 | 450774 |
Speed is implementation-dependent (hardware, batching, tokenizer backend); it is shown here as observed in the benchmark run, not as intrinsic model quality. Token count affects transformer cost later, while tokenizer speed measures only the text-to-ID preprocessing step.
How long each tokenizer’s sequence is compared with the Nepali WordPiece baseline (NSL = 1.0). 1.0 = same as baseline | 3.0 = three times as many tokens | 0.8 = shorter than baseline.
| Tokenizer | NSL avg ↓ | NSL std | Interpretation |
|---|---|---|---|
| MuRIL | 0.82 | 0.11 | Shorter than baseline |
| Nepali WP | 1 | 0 | Baseline |
| Nepali SP | 1.04 | 0.08 | Close to baseline |
| IndicBART | 1.25 | 0.37 | Moderate overhead |
| IndicBERT | 1.28 | 0.38 | Moderate overhead |
| GPT-4 / GPT-3.5 (cl100k) | 3.63 | 1.21 | High overhead |
| GPT-2 | 5.65 | 1.97 | Very high overhead |
MuRIL below 1.0 on NSL means fewer total tokens than the Nepali WP baseline on this corpus, while the leaderboard rank still reflects absolute total tokens (154 vs 183).
Same numbers as the tables above. Each chart uses its own 0–max scale, so compare rows within one chart rather than bar lengths across different charts. For fertility and NSL, shorter is usually better; for chars/token and speed, longer is usually better.
Bar length vs max in corpus · max 9.09
Bar length vs max in corpus · max 3.9
Bar length vs max in corpus · max 5.65
Bar length vs max in corpus · max 466,440
Charts use the same benchmark run as the tables. Speed bars use a distinct tint to remind that raw tok/s depends on hardware and implementation.
These pairs keep the intended meaning roughly constant while changing Nepali register: pronouns and verb morphology move from informal to respectful. A tokenizer is not measuring politeness directly; the chart shows how the changed surface forms affect token count. Vertical axis is one sentence’s token count; horizontal axis is fixed left-to-right: Low → Medium → High → Reverential.
Show tokenizer lines (both charts)
Counts match the visualization section. A flat line means the tokenizer handled these register changes with the same sequence length, not that the forms are linguistically identical.
Nepali WP and SP share the same counts here (7 each); the dashed line traces SentencePiece on top of WordPiece. MuRIL’s count falls because these polite surface forms happen to match larger learned pieces.
Same four honor levels; the question form changes pronoun and verb morphology with register. Colors match the toggles above.
Byte-level BPE models (GPT-2 and cl100k_base) jump at High / Reverential when polite verb forms add characters. Nepali-specific tokenizers stay flat at seven tokens across registers on these two sentences.
Per-sentence token counts aligned with the visualization at the top of the page. Counts include special tokens when the configured tokenizer emits them, because those IDs occupy positions in the sequence. If you change normalization, special-token settings, corpus text, or tokenizer revision, regenerate this table before interpreting the aggregate metrics.
| Corpus row | G2 | cl100k | IB | MuRIL | IBrt | N-WP | N-SP |
|---|---|---|---|---|---|---|---|
| Honor • Low (तँ) — You eat rice | 14 | 11 | 8 | 7 | 7 | 7 | 7 |
| Honor • Medium (तिमी) — You eat rice | 15 | 11 | 7 | 6 | 6 | 7 | 7 |
| Honor • High (तपाईं) — You eat rice | 19 | 15 | 9 | 5 | 8 | 7 | 7 |
| Honor • Reverential (हजुर) — You eat rice | 18 | 15 | 8 | 5 | 7 | 7 | 7 |
| Honor • Low (तँ) — Where are you going? | 14 | 11 | 6 | 6 | 6 | 7 | 7 |
| Honor • Medium (तिमी) — Where are you going? | 15 | 11 | 5 | 5 | 5 | 7 | 7 |
| Honor • High (तपाईं) — Where are you going? | 19 | 15 | 7 | 5 | 7 | 7 | 7 |
| Honor • Reverential (हजुर) — Where are you going? | 18 | 15 | 6 | 5 | 6 | 7 | 7 |
| Simple • Greeting | 22 | 18 | 10 | 6 | 11 | 8 | 10 |
| Simple • Short sentence | 14 | 11 | 7 | 5 | 7 | 7 | 7 |
| Simple • Weather | 15 | 13 | 7 | 5 | 7 | 7 | 7 |
| News • Education policy | 49 | 38 | 19 | 11 | 20 | 13 | 13 |
| News • Traffic | 63 | 48 | 26 | 12 | 24 | 13 | 13 |
| News • Parliament | 44 | 36 | 17 | 9 | 16 | 9 | 11 |
| Tech • AI/ML | 67 | 51 | 29 | 18 | 29 | 21 | 23 |
| Tech • Digital economy | 68 | 54 | 27 | 14 | 26 | 14 | 15 |
| Literary • Himalayas | 77 | 59 | 29 | 16 | 31 | 19 | 21 |
| Literary • Success | 62 | 53 | 26 | 14 | 26 | 16 | 16 |
Hover (or long-press) column headers for full tokenizer names. Totals in overall metrics sum these rows (same preprocessing). Refresh the numbers when your benchmark run or tokenizer setup changes.