Enum TokenizeError 

pub enum TokenizeError {
    InvalidCodepoint(u32),
    MissingByteToken(u8),
    MissingSpecial(&'static str),
}

A text llama.cpp would abort on, by name.

Variants

InvalidCodepoint(u32)

A codepoint above U+10FFFF, which no UTF-8 sequence encodes.

MissingByteToken(u8)

A byte with no byte token in a byte-fallback vocabulary.

MissingSpecial(&'static str)

add_special asked for a beginning- or end-of-sequence token the vocabulary does not have.

Trait Implementations

impl Clone for TokenizeError

fn clone(&self) -> TokenizeError

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizeError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizeError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for TokenizeError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0:

use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0:

replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<InvalidCodepoint> for TokenizeError

fn from(e: InvalidCodepoint) -> Self

Converts to this type from the input type.

impl PartialEq for TokenizeError

fn eq(&self, other: &TokenizeError) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenizeError

impl StructuralPartialEq for TokenizeError