Skip to main content

Module attr

logicaffeine_compile::model::tokenizer

Module attr 

Source
Expand description

A token attribute bit, as llama.cpp’s llama_token_attr.

Constants§

BYTE
CONTROL
LSTRIP
NORMAL
NORMALIZED
RSTRIP
SINGLE_WORD
UNDEFINED
UNKNOWN
UNUSED
USER_DEFINED