Module pre 

The pre-tokenizers of byte-pair vocabularies: which tokenizer.ggml.pre names select which splitter, the regular expressions each one splits a fragment by (in order, each refining the spans of the last), and the vocabulary settings a name implies. The table is llama.cpp’s (llama-vocab.cpp, llm_tokenizer_bpe and llama_vocab::impl::load); model_tokenizer_lock reads both tables out of the pinned source and holds this one equal to them.

Structs

PreSettings

The vocabulary settings a pre-tokenizer name implies, over the byte-pair defaults (clean_spaces on, the rest off). None leaves the default.

Enums

PreType

A byte-pair pre-tokenizer, named as llama.cpp’s LLAMA_VOCAB_PRE_TYPE_*.

Functions

pre_type

The pre-tokenizer a tokenizer.ggml.pre name selects, with the settings it implies; None for a name llama.cpp does not know (it refuses the model). An absent name reads as "default".