Skip to main content

pre_tokenizer_named

Function pre_tokenizer_named 

Source
pub fn pre_tokenizer_named(digest: &str) -> Option<&'static str>
Expand description

The pre-tokenizer the converter’s table names for a probe digest: its last entry for the digest, as the converter’s chain of ifs leaves it when a digest is listed twice.