pub enum PreType {
Show 59 variants
Default,
Llama3,
DeepseekLlm,
DeepseekCoder,
Falcon,
Mpt,
Starcoder,
Gpt2,
Refact,
CommandR,
Stablelm2,
Qwen2,
Qwen35,
Olmo,
Dbrx,
Smaug,
Poro,
Chatglm4,
Viking,
Jais,
Jais2,
Tekken,
Smollm,
Codeshell,
Bloom,
Gpt3Finnish,
Exaone,
ExaoneMoe,
Chameleon,
Minerva,
Deepseek3Llm,
Gpt4o,
GraniteEmbMulti,
TinyAya,
Superbpe,
Trillion,
GraniteDocling,
Bailingmoe,
SeedCoder,
Hunyuan,
HunyuanDense,
HyV4,
JoyaiLlm,
KimiK2,
Ufakzeka,
Grok2,
Afmoe,
Laguna,
MinimaxM2,
SolarOpen,
Mellum2,
K2Horizon,
Spark2_5,
Youtu,
Gemma4,
SarvamMoe,
Mmbert,
Minicpm5,
Whitespace,
}Expand description
A byte-pair pre-tokenizer, named as llama.cpp’s LLAMA_VOCAB_PRE_TYPE_*.
Variants§
Default
Llama3
DeepseekLlm
DeepseekCoder
Falcon
Mpt
Starcoder
Gpt2
Refact
CommandR
Stablelm2
Qwen2
Qwen35
Olmo
Dbrx
Smaug
Poro
Chatglm4
Viking
Jais
Jais2
Tekken
Smollm
Codeshell
Bloom
Gpt3Finnish
Exaone
ExaoneMoe
Chameleon
Minerva
Deepseek3Llm
Gpt4o
GraniteEmbMulti
TinyAya
Superbpe
Trillion
GraniteDocling
Bailingmoe
SeedCoder
Hunyuan
HunyuanDense
HyV4
JoyaiLlm
KimiK2
Ufakzeka
Grok2
Afmoe
Laguna
MinimaxM2
SolarOpen
Mellum2
K2Horizon
Spark2_5
Youtu
Gemma4
SarvamMoe
Mmbert
Minicpm5
Whitespace
Implementations§
Source§impl PreType
impl PreType
Sourcepub fn llama_name(self) -> &'static str
pub fn llama_name(self) -> &'static str
llama.cpp’s name for the type, without the LLAMA_VOCAB_PRE_TYPE_ prefix.
Sourcepub fn regexes(self) -> &'static [&'static str]
pub fn regexes(self) -> &'static [&'static str]
The patterns that split a fragment, applied in order.
Sourcepub fn byte_encode(self) -> bool
pub fn byte_encode(self) -> bool
Whether a split word is written in GPT-2’s byte alphabet before merging (raw UTF-8 otherwise).
Trait Implementations§
impl Copy for PreType
impl Eq for PreType
impl StructuralPartialEq for PreType
Auto Trait Implementations§
impl Freeze for PreType
impl RefUnwindSafe for PreType
impl Send for PreType
impl Sync for PreType
impl Unpin for PreType
impl UnsafeUnpin for PreType
impl UnwindSafe for PreType
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> Downcast for Twhere
T: Any,
impl<T> Downcast for Twhere
T: Any,
§fn into_any(self: Box<T>) -> Box<dyn Any>
fn into_any(self: Box<T>) -> Box<dyn Any>
Convert
Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>. Box<dyn Any> can
then be further downcast into Box<ConcreteType> where ConcreteType implements Trait.§fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
Convert
Rc<Trait> (where Trait: Downcast) to Rc<Any>. Rc<Any> can then be
further downcast into Rc<ConcreteType> where ConcreteType implements Trait.§fn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
Convert
&Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &Any’s vtable from &Trait’s.§fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
Convert
&mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &mut Any’s vtable from &mut Trait’s.§impl<T> DowncastSync for T
impl<T> DowncastSync for T
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Compare self to
key and return true if they are equal.