Module regex 

The regular expressions of the pre-tokenizers that llama.cpp runs through std::regex / std::wregex (unicode_regex_split_stl), matched with the semantics libstdc++ gives them: ECMAScript grammar, depth-first backtracking, the first match in priority order (left alternative first, greedy quantifiers longest first), and the regex_iterator walk over one span — an empty match is retried at its end as non-null and continuous, and every later search runs with match_prev_avail.

A pattern runs over code units of one Width: Width::Narrow is llama.cpp’s collapsed text (one byte per codepoint, a Unicode category folded to one byte, matched by std::regex over signed char), Width::Wide is its codepoint text (std::wregex). The character classes are libstdc++’s in the classic locale, measured against it for every code unit by model_tokenizer_lock: \s is \t \n \v \f \r and space, \d is ASCII digits, \w is ASCII alphanumerics and _, and . is every unit but \n and \r (and, wide, U+2028/U+2029).

The matcher keeps its own backtrack stack, so a long input cannot exhaust the thread’s stack. A quantified group that can match the empty string is refused at compile time: libstdc++ admits one empty iteration where ECMAScript admits none, and no pre-tokenizer pattern depends on either.

Structs

Regex

A compiled pre-tokenizer pattern.

RegexError

A pattern the compiler refuses, by name.

Enums

Width

The code-unit width a pattern runs over.