Module tokenizer 

The tokenizer a GGUF checkpoint carries, read from its tokenizer.ggml.* metadata and run the way llama.cpp runs it (src/llama-vocab.cpp at the pinned commit): text to token ids, ids to text pieces, and the special tokens that end generation.

Two vocabularies are tokenized: byte-pair (gpt2 and its gemma4, hybriddna and whitespace forms — Qwen, Llama 3, gpt-oss, DeepSeek) and SentencePiece (llama — Gemma 3, Llama 2, Mistral). WordPiece, Unigram, RWKV and PLaMo vocabularies are refused by name. model_tokenizer_lock holds every result equal to llama.cpp’s own, on its bundled vocabulary corpora and on seeded random text.

Text is bytes: a fragment that is not UTF-8 tokenizes as llama.cpp tokenizes it (an invalid byte reads as U+FFFD in byte-pair splitting, and SentencePiece falls back to byte tokens).

Modules

attr

A token attribute bit, as llama.cpp’s llama_token_attr.

cxx

The libstdc++ behaviors llama.cpp’s vocabulary inherits, ported step for step.

pre

The pre-tokenizers of byte-pair vocabularies: which tokenizer.ggml.pre names select which splitter, the regular expressions each one splits a fragment by (in order, each refining the spans of the last), and the vocabulary settings a name implies. The table is llama.cpp’s (llama-vocab.cpp, llm_tokenizer_bpe and llama_vocab::impl::load); model_tokenizer_lock reads both tables out of the pinned source and holds this one equal to them.

regex

The regular expressions of the pre-tokenizers that llama.cpp runs through std::regex / std::wregex (unicode_regex_split_stl), matched with the semantics libstdc++ gives them: ECMAScript grammar, depth-first backtracking, the first match in priority order (left alternative first, greedy quantifiers longest first), and the regex_iterator walk over one span — an empty match is retried at its end as non-null and continuous, and every later search runs with match_prev_avail.

split

Pre-tokenization: a fragment of text split into words before merging, as llama.cpp’s unicode_regex_split does.

unicode

The codepoint facts the tokenizers read, as llama.cpp computes them (src/unicode.cpp): category flags, the lowercase map, the lenient UTF-8 decoder (an invalid byte decodes to U+FFFD and advances one byte), and GPT-2’s byte-to-codepoint map. The tables in unicode_tables are generated from llama.cpp’s running code and held equal to it for every codepoint by model_tokenizer_lock.

Structs

Specials

The special token ids, each None when the vocabulary has none.

TokenData

One vocabulary entry.

Vocab

A loaded vocabulary and its tokenizer.

Enums

TokenizeError

A text llama.cpp would abort on, by name.

VocabError

A vocabulary that cannot be read, by name.

VocabKind

The tokenizer family a vocabulary runs.