pub fn tiktoken_split(pattern: &str, text: &str) -> Result<Vec<String>, String>
The pieces tiktoken’s pattern pattern finds in text, in order.
tiktoken
pattern
text