Struct Vocab 

pub struct Vocab {
    pub kind: VocabKind,
    pub model: String,
    pub specials: Specials,
    pub linefeed: Option<u32>,
    pub add_space_prefix: bool,
    pub add_bos: bool,
    pub add_eos: bool,
    pub ignore_merges: bool,
    pub clean_spaces: bool,
    pub escape_whitespaces: bool,
    pub treat_whitespace_as_suffix: bool,
    pub lowercase: bool,
    /* private fields */
}

A loaded vocabulary and its tokenizer.

Fields

kind: VocabKind

model: String

specials: Specials

linefeed: Option<u32>

add_space_prefix: bool

add_bos: bool

add_eos: bool

ignore_merges: bool

clean_spaces: bool

escape_whitespaces: bool

treat_whitespace_as_suffix: bool

lowercase: bool

Implementations

impl Vocab

pub fn from_gguf(g: &Gguf) -> Result<Vocab, VocabError>

Read the vocabulary of g.

pub fn from_meta( meta: &BTreeMap<String, MetaValue>, ) -> Result<Vocab, VocabError>

Read the vocabulary a checkpoint’s metadata describes: its tokenizer.* keys, and the general.* keys whose names llama.cpp keys token attributes by.

pub fn n_tokens(&self) -> usize

pub fn token(&self, id: u32) -> &TokenData

pub fn text_to_token(&self, text: &[u8]) -> Option<u32>

pub fn is_eog(&self, id: u32) -> bool

Whether id ends generation.

pub fn eog(&self) -> impl Iterator<Item = u32> + '_

The end-of-generation tokens, ascending.

pub fn special_order(&self) -> &[u32]

The special tokens in the order fragments are split on them.

pub fn byte_to_token(&self, b: u8) -> Result<u32, TokenizeError>

byte_to_token: the token standing for one byte.

pub fn tokenize( &self, text: &[u8], add_special: bool, parse_special: bool, ) -> Result<Vec<u32>, TokenizeError>

llama_tokenize: the token ids of text. add_special adds the vocabulary’s beginning- and end-of-sequence tokens as it asks; parse_special lets control tokens written in the text match as themselves.

pub fn token_to_piece(&self, id: u32, lstrip: usize, special: bool) -> &[u8]

llama_token_to_piece: the text of id, up to lstrip leading spaces removed; a control or unknown token renders only when special is on.

pub fn detokenize( &self, tokens: &[u32], remove_special: bool, unparse_special: bool, ) -> Vec<u8>

llama_detokenize: the text of tokens. remove_special drops the leading BOS and trailing EOS the vocabulary adds; unparse_special renders control tokens.

Trait Implementations

impl Clone for Vocab

fn clone(&self) -> Vocab

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Vocab

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.