pub fn pre_type(name: &str) -> Option<(PreType, PreSettings)>Expand description
The pre-tokenizer a tokenizer.ggml.pre name selects, with the settings it implies; None for a name llama.cpp
does not know (it refuses the model). An absent name reads as "default".