Module unicode 

The codepoint facts the tokenizers read, as llama.cpp computes them (src/unicode.cpp): category flags, the lowercase map, the lenient UTF-8 decoder (an invalid byte decodes to U+FFFD and advances one byte), and GPT-2’s byte-to-codepoint map. The tables in unicode_tables are generated from llama.cpp’s running code and held equal to it for every codepoint by model_tokenizer_lock.

The flags kept are the ones a tokenizer reads: the category and whitespace bits. llama.cpp’s case and NFD bits never change a split (every codepoint carries exactly one category bit, so a flag word is non-zero either way).

Structs

Flags

A codepoint’s flags, bit for bit llama.cpp’s unicode_cpt_flags.

InvalidCodepoint

A codepoint no UTF-8 sequence encodes: llama.cpp’s unicode_cpt_to_utf8 throws on it.

Constants

MAX_CODEPOINTS

The number of codepoints the flag table covers; a larger value reads as undefined.

Functions

byte_to_utf8

unicode_byte_to_utf8: the UTF-8 of the codepoint byte b is written as.

cpts_from_utf8

unicode_cpts_from_utf8: every codepoint of text, an invalid byte reading as U+FFFD.

flags

The flags of cpt; past the table every codepoint is undefined. A table entry packs a range’s first codepoint above its nine flag bits.

flags_in

The flags of the codepoint at pos of cpts[lo..hi], or none outside it (_get_flags in the splitters).

is_han

unicode_cpt_is_han: the CJK ideograph blocks the Kimi-K2 pre-tokenizer splits on.

len_utf8

unicode_len_utf8: the sequence length a lead byte announces (a continuation byte counts as one).

push_utf8

unicode_cpt_to_utf8, appending to out. Surrogates encode as their three bytes.

to_lower

unicode_tolower: the simple lowercase mapping, or the codepoint itself. The map is stored as runs (first, count, stride, delta): first + k·stride lowercases to itself plus delta for k < count.

utf8_to_byte

unicode_utf8_to_byte: the byte a byte-level codepoint stands for, when it is one.