Module split 

Pre-tokenization: a fragment of text split into words before merging, as llama.cpp’s unicode_regex_split does.

The patterns apply in order, each refining the spans the last one left (spans count codepoints). A pattern llama.cpp has a hand-written splitter for runs that splitter (Custom); every other runs through super::regex, over the collapsed text when it names a Unicode category (\p{L} …, each non-ASCII codepoint folded to one byte standing for its category) and over the codepoints otherwise. A word of a byte-encoded vocabulary is then written in GPT-2’s byte alphabet.

Structs

Splitter

A compiled pre-tokenizer: its patterns, in order, and whether words are byte-encoded.

Enums

Custom

The hand-written splitters, each standing for the pattern llama.cpp dispatches to it.