Struct PreSettings 

pub struct PreSettings {
    pub clean_spaces: Option<bool>,
    pub ignore_merges: Option<bool>,
    pub add_bos: Option<bool>,
    pub escape_whitespaces: Option<bool>,
    pub no_bos_token: bool,
    pub no_lowercase: bool,
}

The vocabulary settings a pre-tokenizer name implies, over the byte-pair defaults (clean_spaces on, the rest off). None leaves the default.

Fields

clean_spaces: Option<bool>

ignore_merges: Option<bool>

add_bos: Option<bool>

escape_whitespaces: Option<bool>

no_bos_token: bool

The name clears the default beginning-of-sequence token (chatglm-bpe, glm4, glm5).

no_lowercase: bool

The name clears the lowercase normalizer (whitespace).

Trait Implementations

impl Clone for PreSettings

fn clone(&self) -> PreSettings

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PreSettings

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for PreSettings

fn default() -> PreSettings

Returns the “default value” for a type.

impl PartialEq for PreSettings

fn eq(&self, other: &PreSettings) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for PreSettings

impl Eq for PreSettings

impl StructuralPartialEq for PreSettings