Struct Tiktoken 

pub struct Tiktoken { /* private fields */ }

A tiktoken tokenizer read from its directory.

Implementations

impl Tiktoken

pub fn open(dir: &Path) -> Result<Tiktoken>

The tiktoken tokenizer of dir: tiktoken.model, the pattern and reserved count its tokenizer code states, and the special tokens tokenizer_config.json names.

pub fn tokens(&self) -> &[(Vec<u8>, u32)]

The mergeable tokens, in the file’s order, with their ranks.

pub fn specials(&self) -> &[(String, u32)]

The special tokens’ text and ids, in id order.

pub fn encode(&self, text: &str) -> Vec<u32>

text encoded as the tokenizer’s encode(text) encodes it, its special tokens allowed.

pub fn merge_of(&self, token: &[u8], rank: u32) -> Option<(Vec<u8>, Vec<u8>)>

The pair byte-pair merging below rank leaves of token (QwenModel.bpe(ranks, token, max_rank=rank)), when it leaves two parts.