Skip to main content

PreType

Enum PreType 

Source
pub enum PreType {
Show 59 variants Default, Llama3, DeepseekLlm, DeepseekCoder, Falcon, Mpt, Starcoder, Gpt2, Refact, CommandR, Stablelm2, Qwen2, Qwen35, Olmo, Dbrx, Smaug, Poro, Chatglm4, Viking, Jais, Jais2, Tekken, Smollm, Codeshell, Bloom, Gpt3Finnish, Exaone, ExaoneMoe, Chameleon, Minerva, Deepseek3Llm, Gpt4o, GraniteEmbMulti, TinyAya, Superbpe, Trillion, GraniteDocling, Bailingmoe, SeedCoder, Hunyuan, HunyuanDense, HyV4, JoyaiLlm, KimiK2, Ufakzeka, Grok2, Afmoe, Laguna, MinimaxM2, SolarOpen, Mellum2, K2Horizon, Spark2_5, Youtu, Gemma4, SarvamMoe, Mmbert, Minicpm5, Whitespace,
}
Expand description

A byte-pair pre-tokenizer, named as llama.cpp’s LLAMA_VOCAB_PRE_TYPE_*.

Variants§

§

Default

§

Llama3

§

DeepseekLlm

§

DeepseekCoder

§

Falcon

§

Mpt

§

Starcoder

§

Gpt2

§

Refact

§

CommandR

§

Stablelm2

§

Qwen2

§

Qwen35

§

Olmo

§

Dbrx

§

Smaug

§

Poro

§

Chatglm4

§

Viking

§

Jais

§

Jais2

§

Tekken

§

Smollm

§

Codeshell

§

Bloom

§

Gpt3Finnish

§

Exaone

§

ExaoneMoe

§

Chameleon

§

Minerva

§

Deepseek3Llm

§

Gpt4o

§

GraniteEmbMulti

§

TinyAya

§

Superbpe

§

Trillion

§

GraniteDocling

§

Bailingmoe

§

SeedCoder

§

Hunyuan

§

HunyuanDense

§

HyV4

§

JoyaiLlm

§

KimiK2

§

Ufakzeka

§

Grok2

§

Afmoe

§

Laguna

§

MinimaxM2

§

SolarOpen

§

Mellum2

§

K2Horizon

§

Spark2_5

§

Youtu

§

Gemma4

§

SarvamMoe

§

Mmbert

§

Minicpm5

§

Whitespace

Implementations§

Source§

impl PreType

Source

pub const ALL: [PreType; 59]

Every type, in declaration order.

Source

pub fn llama_name(self) -> &'static str

llama.cpp’s name for the type, without the LLAMA_VOCAB_PRE_TYPE_ prefix.

Source

pub fn regexes(self) -> &'static [&'static str]

The patterns that split a fragment, applied in order.

Source

pub fn byte_encode(self) -> bool

Whether a split word is written in GPT-2’s byte alphabet before merging (raw UTF-8 otherwise).

Trait Implementations§

Source§

impl Clone for PreType

Source§

fn clone(&self) -> PreType

Returns a duplicate of the value. Read more
1.0.0 · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for PreType

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Hash for PreType

Source§

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher. Read more
1.3.0 · Source§

fn hash_slice<H>(data: &[Self], state: &mut H)
where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher. Read more
Source§

impl PartialEq for PreType

Source§

fn eq(&self, other: &PreType) -> bool

Tests for self and other values to be equal, and is used by ==.
1.0.0 · Source§

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
Source§

impl Copy for PreType

Source§

impl Eq for PreType

Source§

impl StructuralPartialEq for PreType

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
§

impl<T> Downcast for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

Convert Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>. Box<dyn Any> can then be further downcast into Box<ConcreteType> where ConcreteType implements Trait.
§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

Convert Rc<Trait> (where Trait: Downcast) to Rc<Any>. Rc<Any> can then be further downcast into Rc<ConcreteType> where ConcreteType implements Trait.
§

fn as_any(&self) -> &(dyn Any + 'static)

Convert &Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &Any’s vtable from &Trait’s.
§

fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)

Convert &mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot generate &mut Any’s vtable from &mut Trait’s.
§

impl<T> DowncastSync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Send + Sync>

Convert Arc<Trait> (where Trait: Downcast) to Arc<Any>. Arc<Any> can then be further downcast into Arc<ConcreteType> where ConcreteType implements Trait.
§

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

§

fn equivalent(&self, key: &K) -> bool

Checks if this value is equivalent to the given key. Read more
§

impl<Q, K> Equivalent<K> for Q
where Q: Eq + ?Sized, K: Borrow<Q> + ?Sized,

§

fn equivalent(&self, key: &K) -> bool

Compare self to key and return true if they are equal.
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,