pub fn tokenize_for_ui(input: &str) -> Vec<TokenInfo>Expand description
Tokenizes input text and returns token information for UI display.
Each token includes its byte position, text content, and syntactic category for syntax highlighting in the browser interface.