Skip to main content

cpts_from_utf8

logicaffeine_compile::model::tokenizer::unicode

Function cpts_from_utf8 

Source 
pub fn cpts_from_utf8(text: &[u8]) -> Vec<u32>
Expand description

unicode_cpts_from_utf8: every codepoint of text, an invalid byte reading as U+FFFD.