Skip to main content

is_han

logicaffeine_compile::model::tokenizer::unicode

Function is_han 

Source 
pub fn is_han(cpt: u32) -> bool
Expand description

unicode_cpt_is_han: the CJK ideograph blocks the Kimi-K2 pre-tokenizer splits on.